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FOREWORD 


The Software Engineering Laboratory (SEL) is an organization 
sponsored by the National Aeronautics and Space Administration/ 
Goddard Space Flight Center (NASA/GSFC) and created for the 
purpose of investigating the effectiveness of software engi- 
neering technologies when applied to the development of 
applications software. The SEL was created in 1977 and has 
three primary organization members: 

NASA/GSFC (Systems Development and Analysis Branch) 

The University of Maryland (Computer Sciences Department) 

Computer Sciences Corporation (Flight Systems Operation) 

The goals of the SEL are (1) to understand the software de- 
velopment process in the GSFC environment; (2) to measure 
the effect of various methodologies, tools, and models on 
this process; and (3) to identify and then to apply success- 
ful development practices. The activities, findings, and 
recommendations of the SEL are recorded in the Software En- 
gineering Laboratory Series, a continuing series of reports 
that includes this document. A version of this document was 
also issued as Computer Sciences Corporation document 
CSC/TM-85/6017. 
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ABSTRACT 


This document describes a controlled experiment performed by 
the Software Engineering Laboratory (SEL) to compare the 
effectiveness of code reading, functional testing, and 
structural testing as software verification techniques. It 
is one of a series of three experiments organized by 
R. W. Selby as part of his doctoral dissertation. The 
experiment results indicate that code reading provides the 
greatest error detection capability at the lowest cost, 
whereas structural testing is the least effective tech- 
nique. This document explains the experiment plan, de- 
scribes the experiment results, and discusses related 
results from other studies. It also considers the applica- 
tion of these results to the development of software in the 
flight dynamics environment. Appendixes summarize the ex- 
periment data and list the test programs. A separate Data 
Supplement contains original materials collected from the 
participants. 
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SECTION 1 


INTRODUCTION 


Probably less work, both theoretical and practical, has been 
invested in the development of effective tools, practices, 
and techniques for testing than for any other phase of the 
software life cycle. Many of the innovations suggested have 
been organizational rather than methodological, for example, 
independent verification and validation (Reference 1) , 
clean-room development (Reference 2) , and separate test 
teams (Reference 3) . Decisions about the organization of 
testing activities are premature, however, when the relative 
effectiveness of different testing methodologies has not yet 
been established. 

The principal methodological approaches to software testing 
and verification are code reading (Reference 4) , functional 
testing (Reference 5) , and structural testing (Refer- 
ence 5) . Most experts recommend a mix of these techniques 
(e.g.. Reference 6); the relative merits of these very dif- 
ferent approaches are not, however, well understood. 

This document describes an experiment (Reference 7) per- 
formed by the Software Engineering Laboratory (SEL) to com- 
pare the effectiveness of these three techniques. It is one 
of a series of experiments organized by R. W. Selby as part 
of his doctoral dissertation (Reference 8) at the University 
of Maryland. The results of the experiment were examined to 
determine how the use of code reading, functional testing, 
and structural testing can best be organized in the flight 
dynamics environment (where they are already applied to dif- 
fering degrees) . 

1.1 SOFTWARE VERIFICATION EXPERIMENT 

The SEL planned a controlled experiment to compare the ef- 
fectiveness of code reading, functional testing, and struc- 
tural testing. During a 1-month period, over 40 professional 
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programmers participated in the screening and other phases 
of the experiment. Subjects completed a background survey 
and a pretest prior to the actual experiment. The experi- 
ment consisted of three software testing sessions. This 
document presents an analysis and discussion of the experi- 
ment, describes the test programs, and summarizes the exper- 
iment data. The Data Supplement contains the original 
materials collected from the participants. 

1.2 SOFTWARE ENGINEERING LABORATORY 

The SEL conducted the experiment described in this docu- 
ment. The SEL is a research project sponsored by Goddard 
Space Flight Center (GSFC) and supported by Computer 
Sciences Corporation and the University of Maryland (Refer- 
ence 9) . The overall objective of the SEL is to understand 
the software development process at GSFC and to identify 
ways in which it can be modified to improve the quality and 
reduce the cost of the product. 

The SEL collects and analyzes data from software development 
projects that support flight dynamics activities at GSFC. 
Measures collected include staffing, computer utilization, 
error reports, and product size/complexity statistics, as 
well as the technologies applied. More than 40 projects 
have been monitored by the SEL during the past 8 years. SEL 
principals also participated in the management of these 
projects. The data collected from these projects have been 
assembled in the SEL computer data base. Reference 9 de- 
scribes the SEL and its activities in more detail. 

Three types of experiments have been performed by the SEL: 
screening, semicontrolled , and controlled. Screening exper- 
iments provide detailed information about how software is 
currently developed in the environment under study. No at- 
tempt is made to impose new or different methodologies on 
these tasks. In semicontrolled experiments, on the other 
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hand, specific methodologies are designated' for application, 
reinforced by training and management direction. Controlled 
experiments can be implemented in either of two ways. Two 
(or more) carefully matched individuals (or teams) may be 
assigned the same task but required to use different method- 
ologies. Alternatively, the teams (or individuals) may not 
be matched but instead be assigned consecutive tasks, apply- 
ing different methodologies to each task. 

Screening and semicontrolled experiments employ production 
flight dynamics projects; however, the high cost of full- 
scale development makes controlled experiments with produc- 
tion projects impractical. The testing experiment described 
in this document represents the next best alternative, a 
controlled experiment using production programmers. 

1.3 FLIGHT DYNAMICS ENVIRONMENT 

The general class of spacecraft flight dynamics software 
studied by the SEL includes ground-based applications to 
support attitude determination, attitude control, maneuver 
planning, orbit adjustment, and mission analysis. System 
sizes range from 33 to 159 thousand lines of source code, 
and development schedules range from 13 to 21 months. The 
fixed spacecraft launch date requires close adherence to 
schedule. Table 1-1 summarizes the characteristics of 
flight dynamics software. 

Most flight dynamics projects are developed on a group of 
IBM mainframe computers using FORTRAN and assembly lan- 
guages. Smaller projects are developed on a VAX super mini- 
computer. Some support software is developed on a PDP 
minicomputer. Remote terminals provide easy access to all 
computers. Data are collected from flight dynamics projects 
via questionnaires, computer accounting, automated tools, 
and management reviews. Reference 9 describes the flight 
dynamics environment in more detail. 
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Table 1-1. Characteristics of Flight Dynamics Environment 


Process Characteristics 

Av 

High 

Low 

Duration (months) 

16 

21 

13 

Effort (staff years) 

8 

24 

2 

Size (1000 source lines of code) 

Developed 

57 

142 

22 

Delivered 

62 

159 

33 

Staff (full-time equivalent) 

Average 

5 

11 

2 

Peak 

10 

24 

4 

Individuals 

14 

29 

7 

Application experience (years) 

Managers 

6 

7 

5 

Technical staff 

4 

5 

3 

Overall experience (years) 

Managers 

10 

14 

8 

Technical staff 

9 

11 

7 


NOTES : Type of Software: Scientific, ground-based, inter- 

active graphic. 

Languages: 85 percent FORTRAN, 15 percent assembler 

macros . 

Computers: IBM, PDP, VAX. 
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Figure 1-1 shows the distribution of effort by software life 
cycle activity in the flight dynamics environment. As indi- 
cated in the figure, testing consumes a substantial portion 
(up to 40 percent) of development resources. Any method of 
increasing the efficiency and effectiveness of testing ac- 
tivities would be welcomed. The purpose of the experiment 
described in this report was to compare three relevant 
software test/verification techniques and determine how they 
can best be integrated into flight dynamics software devel- 
opment practice. 
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SECTION 2 - EXPERIMENTAL APPROACH 


This section describes the three software verification tech- 
niques evaluated in the experiment as well as the experimen- 
tal procedure followed and the statistical design employed 
in the analysis of the experiment data. It also includes 
some observations on the way the experiment was conducted. 

2 . 1 VERIFICATION TECHNIQUES 

Many different variations of the code reading, functional 
testing, and structural testing approaches are possible. 

For purposes of this experiment, one specific variation of 
each was selected. The three techniques differ with respect 
to tne access they provide to information about the program 
tested (see Table 2-1) . Consequently, the techniques differ 
witn respect to who can apply them effectively. For exam- 
ple, users and analysts can perform functional testing be- 
cause it does not require studying the source code, but they 
would probably not be very successful with code reading or 
structural testing. The following subsections define the 
specific versions of these techniques that the experimenters 
encouraged the subjects to use. 


Table 2-1. Characteristics of Verification Techniques 


Code 

Characteristic Reading 


View program Yes 

specification 

View source Yes 

code 

Execute No 

program 


Functional 

Testing 

Yes 

No b 


Structural 

Testing 

No a 

Yes 


Yes Yes 


a Specif ication was supplied after all tests were executed. 
^Source code was supplied after all tests were executed. 
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2.1.1 CODE READING 


Code reading is a systematic procedure for reading and un- 
derstanding the operation of a program. Developers read 
code to determine if a program is correct with respect to a 
given function. Techniques of code reading include check- 
lists, simulated execution, and step-wise abstraction (Ref- 
erence 4) . in practice, developers employ some aspects of 
all three techniques. For this experiment, subjects were 
trained in and encouraged to use the method of step-wise 
atstr action . 

The method of step-wise abstraction is based on the concepts 
of proper subprograms, prime programs, and structured pro- 
grams. Large structured programs are made up of smaller 
ones (subprograms) . Those subprograms that cannot be fur- 
ther decomposed are prime programs. Identifying the prime 
programs in a software segment and then combining them to 
form higher levels of abstraction allows the actual function 
of the software to be defined and any errors or inconsis- 
tencies to be recognized (Reference 4) . During the experi- 
ment, subjects attempted to recognize prime programs in the 
source code and abstract their functions. 

2.1.2 FUNCTIONAL TESTING 

Functional testing is a strategy for designing and selecting 
test cases. It treats the software like a "black box." 

Input is supplied, and output is observed. Comparison of 
the software specification with the observed input/output 
relationship indicates whether an error is present in the 
software. Functional testing does not require the tester to 
have any knowledge of the design or operation of the soft- 
ware . 

For functional testing to be complete, all possible input 
must be tested. Clearly, this is not practical for a pro- 
gram of any significant size. One approach is to select 
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at random from among the possible test cases (inputs). How- 
ever, the experimenters selected the alternative strategy of 
equivalence partitioning for study. Equivalence partition- 
ing also reduces the amount of input that must be tested to 
have reasonable confidence in the system. This is done by 
dividing each input condition (usually a statement in the 
specification) into two or more groups (Reference 5) . Both 
valid and invalid equivalence classes must be tested. After 
testing was completed, subjects were provided with the 
source code to isolate and correct the errors found. 

2.1.3 STRUCTURAL TESTING 

Structural testing is another strategy for designing and 
selecting test cases. As opposed to functional testing, it 
treats the software like a "white box." Tests are specified 
based on an examination of the software structure (rather 
than the specification) . Structural testing compares the 
detailed system design to its software implementation. 
Ideally, structural tests should be based on the program 
design language (PDL) and developed at the same time as the 
PDL. Structural tests alone do not provide a mechanism for 
verifying the software against the specification. 

Coverage (the degree to which the software is exercised) 
serves as the basic criteria for completion of structural 
testing (Reference 5) . Three levels of coverage are recog- 
nized: Statement coverage requires that every statement in 

the source code be executed at least once. Condition (or 
branch) coverage requires that each outcome of every deci- 
sion be executed at least once. Path coverage requires that 
every possible sequence of decision outcomes be executed, 
whicn can lead to an impr actically large number of tests for 
a program of any significant size or complexity. For exam- 
ple, backward transfers can produce an infinite number of 
paths. 
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Statement coverage is the weakest form of coverage in the 
sense that it can generally be satisfied with the fewest 
test cases. It is, however, the form of structural testing 
evaluated in this experiment. Subjects were directed to 
stop testing after 100-percent statement coverage had been 
achieved. After testing was completed, subjects were pro- 
vided with the specification to compare against test results. 

2.2 EXPERIMENT PROCEDURE 

The experiment proceeded in three phases: experiment prepa- 

ration, experiment execution, and data analysis. The fol- 
lowing subsections describe these phases and present some 
observations on the way the experiment was conducted. 

2.2.1 PREPARATION 

During the month prior to the start of the experiment ses- 
sions, several preparatory steps were taken: 

• Subject selection 

• Preexperiment survey 

• Training session 

• Experiment pretest 

• Environment setup 

The experimenters screened about 50 professional programmers 
to select an appropriate sample of 32 for the planned sta- 
tistical design (Section 2.2.3). Subjects were selected to 
represent three different levels of expertise defined by 
professional experience and educational background. Sub- 
jects were also about equally divided between IBM and VAX 
users. 

All experiment candidates completed an extensive question- 
naire describing their education and relevant experience. 
Subject selection was based on questionnaire responses as 
well as information supplied by managers. The Data Supple- 
ment contains the original questionnaire materials. 
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Subjects participated in a 3-hour tutorial explaining the 
three software verification techniques. Most subjects had 
previous experience with functional testing and informal 
code reading. Most, however, lacked previous formal train- 
ing in any of the three techniques. 

After the training session, subjects completed an experiment 
pretest. The subjects solved problems and answered ques- 
tions defining their understanding of the three techniques 
and their attitude toward the experiment. The Data Supple- 
ment contains the original pretest materials. 

Each subject used either an IBM 4341 or VAX 11/780 computer 
to perform both functional and structural testing. Before 
the experiment sessions began, separate versions of the 
three test programs were developed for each computer. Most 
of the differences between the two versions are minor. Ap- 
pendix A provides complete source listings for the VAX ver- 
sion. Table 2-2 summarizes the test programs. 

Table 2-2. Test Programs 


Test 

Proqram 

Software 

Type 

Executable 

Statements 

Subroutines 

Faults 

1 

Text formatter 

55 

3 

9 

2 

List manager 

48 

9 

7 

3 

File maintainer 

144 

7 

12 


The experimenters established temporary IDs and special ac- 
counts on each computer. During testing, subjects invoked 
each program via a driver that also recorded the number of 
tests performed and the degree of structural coverage at- 
tained. Operating system accounting information provided 
the connect time and CPU utilization for each subject. 
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2.2.2 EXECUTION 


The actual experiment was conducted in three sessions over a 
2-week period. Although tight, the schedule of experimental 
activity was strictly adhered to. During each experiment 
session, all subjects tested the same program. However, all 
three verification techniques were applied to each program 
as prescribed in the statistical design (Section 2.2.3). 

Each subject used only one technique in any given experiment 
session. Tests were executed on the same computers used by 
the subjects in their usual work. 

Each experiment session was scheduled to be completed in a 
single day. The experimenters felt that the experimental 
tasks could easily be completed within 8 hours. An experi- 
menter distributed the test materials in the morning, then 
collected the results in the afternoon. However, a few sub- 
jects who did not complete testing on the prescribed day due 
to interruptions submitted their results on the following 
day. Each subject returned a list of errors, corrected 
source listing, and estimates of effort expended and faults 
found. An experimenter was available throughout the testing 
period to answer the subjects' questions. 

Code readers received a specifications statement and source 
listing at the start of the experiment session. They were 
not allowed to execute the test program. All errors de- 
tected were found by inspecting the code and performing the 
step-wise abstraction process described in Section 2.1.1. 

Functional testers received a specifications statement and 
access to the program driver at the start of the experiment 
session. They defined tests as described in Section 2.1.2. 
After executing the tests, access to the program driver was 
removed, but a source listing was provided. Subjects sepa- 
rately identified the errors detected during testing and 
those found during source correction. 
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Structural testers received a source listing and access to 
the program driver at the start of the experiment session. 
They defined and executed tests until achieving near 
100-percent statement coverage (Section 2.1.3). After exe- 
cuting the tests, access to the program driver was removed, 
but a specifications statement was provided. Subjects iden- 
tified errors by comparing the test output with the specifi- 
cations . 

2.2.3 ANALYSIS 

The data analysis employed a fractional factorial design 
(Reference 10) . The experimenters divided the subjects into 
three groups on the basis of overall experience. Within 
each group, the sequence of code reading, functional test- 
ing, and structural testing was varied to cover all possible 
combinations about equally. Table 2-3 shows how the sub- 
jects were divided into expertise groups and then assigned 
combinations of verification techniques and programs. 

Because each subject used each technique once and tested 
each program once, rather than experiencing every possible 
combination of technique and program, the design is a frac- 
tional rather than full factorial model. The three programs 
were always presented in the same order, thereby limiting 
the opportunity for subjects who had completed testing a 
program to discuss it with others who had not yet tested 
it. This statistical design enabled the effects of program- 
mer experience and program tested to be eliminated from the 
evaluation of the three verification techniques. It as- 
sumes, however, that there is no interaction between subject 
and technique or program. 

The dependent variables studied were number of faults found, 
effort to detect faults, and effort to correct faults. Sec- 
tion 3 presents the results of this analysis. Although the 
analysis based on the fractional factorial design was 
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Table 2-3. Fractional Factorial Design (Reference 7) 


SUBJECT 

CLASSIFICATION 

CODE 

READING 

FUNCTIONAL 

TESTING 

STRUCTURAL 

TESTING 

P 1 P 2 P 3 

P 1 P 2 p 3 

p 1 p 2 p 3 

Si 

ADVANCED S 2 

• 

• 

• 

s 8 








• • • 

• • • 


A 



S 9 

INTERMEDIATE S 10 

• 

• 

• 

s 19 








. • • 

• • • 





S 20 

JUNIOR S 21 

• 

• 

• 

S 32 








• • • 

• • • 






NOTES: BLOCKING ACCORDING TO EXPERIENCE LEVEL AND PROGRAM TESTED. 
EACH SUBJECT USES EACH TECHNIQUE AND TESTS EACH PROGRAM. 
S x = SUBJECT X. 

P x = PROGRAM X. 
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completed within 2 months of the last experiment session, 
the data collected daring the experiment will enable addi- 
tional analyses to be performed later. 

2.2.4 OBSERVATIONS 

After the completion of the experiment sessions, the sub- 
jects indicated their reactions to the experiment process 
and verification techniques evaluated. Relevant comments 
included the following: 

• A majority of subjects believed functional testing 
to have been the most effective software verifica- 
tion strategy. 

• A majority of subjects did not follow the step-wise 
abstraction method of code reading exactly or ex- 
clusively. 

• Several subjects indicated that satisfaction of the 
100-percent statement coverage criteria caused them 
to stop structural testing, even though they felt 
that more testing was needed. 

• Several subjects disagreed with the experimenter's 
definitions of errors. These conflicts arose from 
differing interpretations of the severity of prob- 
lems as well as specification ambiguity. 

• Some subjects were not able to concentrate fully on 
the experiment due to conflicting responsibilities 
(not unlike conditions obtaining during normal 
software development) . 

• An office relocation that occurred on the same day 
as the last experiment session interrupted the ac- 
tivities of almost all the subjects. 
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Nevertheless, none of the subjects indicated that the over- 
all test results seemed biased or that the experiment proce- 
dure was seriously flawed. 
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SECTION 3 


EXPERIMENT RESULTS 


Thirty-two subjects participated in the three experiment 
sessions. Appendix B documents their performance with each 
verification technique. The data were analyzed with a frac- 
tional factorial design (described in Section 2.2.3). The 
principal areas for comparison among the techniques were as 
follows : 

• Effectiveness of fault detection in terms of the 
number of faults detected 

• Cost of fault detection/correction in terms of the 
effort (in hours) per fault to detect and correct 
the faults identified 

• Types of errors to which each technique was sensi- 
tive 

• Role of subject experience in technique effective- 
ness 

The following subsections describe the results obtained from 
the experiment in each of these areas. 

3.1 EFFECTIVENESS OF FAULT DETECTION 

Table 3-1 lists the average number of errors detected by 
subjects using each technique for each program. Figure 3-1 
summarizes the faults detected by the techniques across pro- 
grams. The figure shows that code reading detected the most 
faults overall. Code reading detected significantly more 
faults than functional testing, and functional testing de- 
tected significantly more faults than structural testing. 

Each program contained a different number of faults. Never- 
theless, analyzing the data in terms of the percentage of 
total faults detected yields the same result (Figure 3-2) . 
Code reading is clearly superior to either online testing 
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technique with respect to fault-detection effectiveness. 
However, the variation due to verification technique appears 
to be much less than that due to the specific program under 
examination. 


Table 3-1. 

Fault 

Detection 

Effectiveness 


Verification 



Program 


Technique 

1 

2 

3 

Overall 

Code reading 

5.5 

6 . 6 

3.2 

5.1 

Functional testing 

4.6 

4.6 

4.2 

4.5 

Structural testing 

2.5 

4.4 

2.8 

3.3 


NOTE: Values are average number of faults detected. 

The difference in fault-detection effectiveness between 
functional and structural testing stands out because these 
techniques are similar in many ways. Both structural and 
functional testing achieved about the same level of state- 
ment coverage (97 percent) , although it was not a criterion 
for functional testing. Furthermore, both structural and 
functional testing exposed about the same percentage of 
faults (62 percent) , but not all faults exposed by testing 
were recognized as such by the subjects. Functional testers 
recognized 19 percent more of the observable faults than did 
structural testers. 

3.2 COST EFFECTIVENESS OF FAULT DETECTION/CORRECTION 

Table 3-2 lists the average number of faults detected and 
corrected per hour of effort using each technique for each 
program. Figure 3-3 summarizes the fault detection/ 
correction rate for each technique across programs. The 
figure shows that code reading detected the most faults per 
hour of effort. Code reading performed significantly better 
than functional or structural testing, which were not 
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FAULTS DETECTED 


5.1 


CODE 

READING 


4.5 


FUNCTIONAL 

TESTING 


3.3 


’ STRUCTURAL 
TESTING 


NOTES: CODE READING > OTHERS. 

FUNCTIONAL > STRUCTURAL. 

PROBABILITY OF DIFFERENCE BEING RANDOM IS LESS THAN 0.005. 


Figure 3-1. 


Number of Faults Detected (Reference 7) 
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significantly different from each other. No significant 
differences, however, exist among the techniques with 
respect to the total time spent looking for and correcting 
faults (see Figure 3-4) . 

Table 3-2. Fault Detection/Correction Rate 


Verification 


Program 


Technique 

1 

2 

3 

Overall 

Code reading 

1.1 

3.6 

0.9 

1.9 

Functional testing 

1.1 

1.2 

0.7 

o 

• 

H 

Structural testing 

1.1 

1.7 

0.5 

1.1 


NOTE ; Values are average number of faults detected and cor- 
rected per hour of effort. 

The experimenters also monitored computer utilization during 
functional and structural testing. (Code reading requires 
no computer resources.) Functional testing expended signif- 
icantly more CPU time than structural testing; however, no 
differences were detected in the number of test runs. 

3.3 CHARACTERISTICS OF FAULTS 

Table 3-3 identifies the classes of faults present in the 
test programs according to two criteria. The action associ- 
ated with a fault identifies whether it is due to omitting 
some necessary code or incorrectly implementing code. The 
location of a fault defines where it occurs in the code. 

Table 3-4 indicates that code reading and functional testing 
proved to be substantially more effective than structural 
testing in detecting faults of omission. Seven such faults 
were detected by less than 25 percent of the structural 
testers, whereas the same proportion of code readers and 
functional testers left only five and four omission faults, 
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respectively, undetected. This result seems reasonable -• 
given that structural tests were based solely on the exist- 
ing code. 


Table 3-3. Fault Characterization (Reference 7) 




Action 


Location 

Omission 


Commission 

Initialization 

0 


2 

Computation 

2 


2 

Control 

2 


4 

Interface 

2 


11 

Data 

2 


0 

Cosmetic 

0 


_1 

TOTAL 

NOTE: Values are 

8 

numbers of faults. 


20 


Table 3-4 also shows structural testing to be less effective 
in detecting faults of commission. Only three such faults 
were detected by 75 percent or more of the structural 
testers, whereas the same proportion of code readers and 
functional testers detected nine and six commission faults, 
respectively . 

The only fault location sufficiently represented in the sam- 
ple to provide a basis for conclusions was interface fault. 
As shown in Table 3-5, two or three such faults were found 
by 75 percent or more of online testers, whereas seven in- 
terface faults were detected by 25 percent or more of code 
readers. Code reading thus appears to be the superior tech- 
nique with respect to interface errors. 
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Table 3-4. Detection of Omission/Commission Faults 
(Reference 7) 


Percent of Subjects Code Functional 

Detecting This Fault Reading Testing 


NOTES : 
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Table 3-5. Detection of Interface Faults (Reference 7) 


Percent of Subjects Code 

Detecting This Fault Reading 


Functional Structural 

Testing Testing 


100 


III I 

II 


I 


75 

50 


I 

I 

25 

I 

I 

0 II 


I 

II 
II 

I 

II 
II 


I- 

II 


I 

II 


I 

I 

I 

III 


NOTES: I = interface fault 


3.4 EFFECT OF SUBJECT EXPERTISE 

The experimenters assigned subjects to three expertise 
levels on the basis of their professional experience and 
educational background. Table 3-6 shows subject performance 
by expertise level. Advanced subjects detected signifi- 
cantly more faults than intermediate or junior subjects. 
However, the detection rate (faults detected per hour of 
effort) did not appear to be related to level of expertise. 

Figures 3-5 and 3-6 show the interaction of expertise level 
with program tested and verification technique, respec- 
tively. The specific program tested appears to have a 
greater effect on the percent of faults found than expertise 
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level does. Also, the advanced subjects performed substan- 
tially better than intermediate or junior subjects when code 
reading or structural testing. The percent of faults found 
when functional testing was similar for all expertise levels. 

Table 3-6. Effect of Subject Expertise 


Expertise 

Number of 

Faults 

Detect io 

Level 

Subjects 

Detected 

Rate 3 

Advanced 

8 

5.0 

2.36 

Intermediate 

11 

4.2 

2.53 

Junior 

13 

3.9 

2.14 


a Faults detected per hour of effort. 
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Figure 3-6. Interaction of Expertise Level and 
Verification Technique 
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SECTION 4 - RELATED RESULTS 


This section reviews the results of the experiment described 
in Sections 2 and 3 in light of related research performed 
by the SEL and others. Specifically, an attempt is made to 
identify the implications of this experiment for developer 
training and development organization. 

4.1 OTHER VERIFICATION TECHNIQUE EVALUATIONS 

The results of this experiment imply that code reading de- 
tects the most faults at the lowest cost. Several relevant 
studies have been conducted at the University of Maryland. 

A prototype study of the same three verification techniques 
by Hwang (Reference 11) suggested that code reading does at 
least as well as the computer-based techniques. Two earlier 
experiments by Selby (Reference 12) using student subjects 
found that functional testing detected the most faults. In 
one of these experiments, code reading demonstrated the 
lowest fault detection cost; functional testing led in the 
other . 

Other researchers, as well, have studied these techniques. 
Card et al. (Reference 13) showed that code reading in- 
creases the reliability of the delivered software product. 

It was not, however, compared with the other techniques. 
Myers (Reference 14) evaluated functional testing and 
three-person code reviews versus a control group. All three 
groups proved to be equally effective in detecting errors, 
but code reviews cost substantially more. Hetzel (Refer- 
ence 15) compared functional testing, code reading, and a 
composite testing technique and found code reading to be 
inferior to the other techniques. 

A significant difference between the experiment reported in 
this document and many other studies is the use of profes- 
sional programmers as subjects. Viewed in this context, the 
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effectiveness of code reading relative to the other tech- 
niques appears to increase with experience. Effective code 
reading may require not only training in the technique of 
step-wise abstraction but also the recognition of common 
programming paradigms. Experiments by Soloway and Ehrlich 
(Reference 16) showed that experienced programmers surpassed 
novices in detecting and correcting faults when certain com- 
mon programming plans and rules of discourse were followed 
in the implementation of the code. The performance of ex- 
perienced programmers approximated that of novices when the 
plans and rules were not followed. 

Together, these results suggest that code reading is most 
effective when performed by individual experienced program- 
mers. On the other hand, computer-based testing can be 
performed effectively by less-experienced programmers. 
Furthermore, functional testing works well for an independ- 
ent test team (who do not have the developers' knowledge of 
the code) . 

4.2! INDEPENDENT VERIFICATION AND VALIDATION 

Independent verification and validation (IV&V) is an ap- 
proach to organizing software development in which an inde- 
pendent team is assigned to verify and validate each 
life-cycle product of the development team (Reference 17) . 
Specifically, the IV&V team performs independent system 
testing in addition to that done by the development team. 
Many of the errors reported by an IV&V team duplicate those 
found by the development team. 

The IV&V team does not usually read code. Given that online 
testing is less efficient than code reading, it is not sur- 
prising that a recent study showed that the principal effect 
of IV&V was to increase development costs in the flight dy- 
namics environment (Reference 18) , where code reading is al- 
ready practiced. IV&V may provide some benefit in the 
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earlier phases of requirements and design, but that is more 
difficult to demonstrate. 

4.3 CLEAN-ROOM DEVELOPMENT 

Dyer and Mills (Reference 2) proposed that software develop- 
ment can and should be done without the aid of computers. 
Instead of computer-based testing, the development team 
should rely on code reading, code inspections, and formal 
verification techniques to identify and correct faults. 

After development (of a build or a system) was complete, an 
independent testing team would certify the reliability of 
the developed software. 

A recent experiment by Selby (Reference 8) showed that, rel- 
ative to a control group, development teams using the clean- 
room approach delivered products that passed a higher 
percentage of test cases, more completely satisfied the sys- 
tem requirements, included more comments, and exhibited 
lower complexity. Another SEL study (Reference 13) indi- 
cated that a high rate of computer use by the development 
team is associated with low productivity. 

The results of the testing experiment reported in this docu- 
ment demonstrate that one clean-room technique, code read- 
ing, surpasses the effectiveness of online testing 
techniques. This evidence, together with the results of the 
clean-room experiment (Reference 8) , indicates that the 
clean-room approach to software development is well founded 
and viable. However, it does not prove its value in every 
circumstance, or specifically in the flight dynamics envi- 
ronment. Further study of this approach is needed. 
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SECTION 5 - CONCLUSIONS 


As described in Section 2, the controlled experiment com- 
pared’ the performance of code reading, functional testing, 
and structural testing on three software verification 
tasks. These techniques were applied by professional pro- 
grammers working in their usual production environment. The 
results of the controlled experiment (Section 3), together 
with the related results presented in Section 4, provide 
considerable guidance about the overall effectiveness of the 
software verification techniques studied and about how to 
best employ them in the flight dynamics environment. 

5. i EVALUATION OF VERIFICATION TECHNIQUES 

Each of the verification techniques showed some merit in the 
controlled experiment. Code reading performed best over- 
all. Functional testing was second. The relatively poor 
showing of structural testing may be due, in part, to the 
weak coverage criterion used (i.e., statement coverage). 

Prior to the experiment, only 22 percent of respondents to 
the background survey believed that code reading was the 
most effective software verification technique. About equal 
proportions (38 percent and 40 percent, respectively) fa- 
vored structural or functional testing. After participating 
in the experiment, the majority of subjects felt that func- 
tional testing had proved to be the most effective technique. 

Nevertheless, subjects applying code reading detected the 
most faults and expended the least effort per fault to do 
so. Functional testing actually proved to be the second 
most effective technique in terms of the number of faults 
detected. Functional testers were more likely to recognize 
an error that was uncovered by testing than were structural 
testers. 
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Both code reading and functional testing detected faults of 
commission and omission better than did structural testing. 
Code reading showed itself to be the most effective tech- 
nique with respect to interface errors. Furthermore, the 
advanced expertise group succeeded substantially better than 
intermediate or junior subjects at code reading effec- 
tively. The results of the experiment also indicated that 
the effectiveness of these verification techniques may de- 
pend on program size. This factor still needs to be inves- 
tigated . 

5.2 APPLICATION IN FLIGHT DYNAMICS ENVIRONMENT 

The results of the controlled experiment and related studies 
suggest several modifications to the process of software 
development in the flight dynamics environment. These in- 
clude the following: 

• Formalize the code reading process. 

• Provide specific instruction in code reading. 

• Assign senior developers to code reading. 

• Assign junior developers to functional testing, 
possibly as an independent test team. 

• Do not apply structural testing. However, monitor- 
ing the coverage achieved may be useful for func- 
tional testing. 

An earlier study of flight dynamics acceptance testing prac- 
tices (Reference 19) supports many of these conclusions. 

That study was based on a review of the relevant literature 
and observation of then-current procedures. Its recommenda- 
tions included the following: acceptance test each build, 

trace acceptance tests to requirements, measure test cover- 
age, use equivalence partitioning (to reduce test cases) , 
perform independent acceptance testing, and record error 
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histories. Many of these recommendations have already been 
implemented . 

Together, these studies demonstrate that the specific tech- 
niques applied, as well as the expertise level of the 
testers, affect the overall rate and cost of fault detection/ 
correction. Selecting and combining the appropriate tech- 
niques, personnel, and organization can make a significant 
contribution to software quality. 
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APPENDIX A 


TEST PROGRAMS 


This appendix reproduces the specifications, source list- 
ings, and error identifications for the test programs used 
in tne experiment. The test programs represent three dif- 
ferent types of software: 

• Text formatter 

• List manager 

• File maintainer 

Congruent versions of these programs were developed for both 
the VAX-11/780 and IBM 4341 computers. The VAX versions are 
supplied in this appendix. The Data Supplement contains 
both versions of the specifications and source listings. 
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1) VAX: Code Reading 


ID# 


Specification 

Given an Input text of up to 80 characters consisting of words separated by blanks 
or new-line characters, the program formats It Into a llne-by-llne form such that i) each 
output line has a maximum of 30 characters, 2) a word In the Input text Is placed on a 
single output line, and 3) each output line Is filled with as many words as possible. 

The Input text Is a stream of characters, where the characters are categorized as 
either break or nonbreak characters. A break character Is a blank, a new-line character 
($), or an end-of-text character (/). New-line characters have no special significance; 
they are treated as blanks by the program. The characters $ and / should not appear In 
the output. 

A word Is defined as a nonempty sequence of nonbreak characters. A break Is a 
sequence of one or more break characters and Is reduced to a single blank character or 
start of a new line In the output. 

When the program Is Invoked, the user types the Input on a single line, followed by 
by a / (end-of-text) and a carriage return. The program then formats the text and 
types It on the terminal. 

If the input text contains a word that Is too long to fit on a single output line, an 
error message Is typed and the program terminates. If the end-of-text character Is miss- 
ing, an error message Is Issued and the program awaits the Input of a properly ter- 
minated line of text. (End of specification.) 


1) VAX: Reading Functional Structural 


ID# 


001 

002 : 

003 

004: 

005 

006 

007 

008 

009 

010 
011 
012 

013 

014 

015 

016 

017 

018 

019 

020 
021 
022 

023 

024 

025 

026 

027 

028 

029 

030 

031 

032 

033 

034 

035 

036 

037 

038 

8 $ 

041 

042 

043 

044 

045 

046 

047 

048 

049 

050 

051 

052 

053 


C NOTE THAT YOU DO NOT NEED TO VERIFY THE FUNCTION 'MATCH'. 

C IT IS DESCRIBED THE FIRST TIME IT IS USED, AND ITS SOURCE CODE 
C IS INCLUDED AT THE END FOR COMPLETENESS. 

C 

C NOTE THAT FORMAT STATEMENTS FOR WRITE STATEMENTS INCLUDE A LEADING 
C AND REQUIRED ' ' FOR CARRIAGE CONTROL 

C VARIABLE USED IN FIRST, BUT NEEDS TO BE INITIALIZED 
INTEGER MOREIN 

C STORAGE USED BY GCHAR 
INTEGER BCOUNT 
CHARACTER* 1 GBUFER(80) 

CHARACTER*80 GBUF 

C GBUFER AND GBUFSTR ARE EQUIVALENCED 

C STORAGE USED BY PCHAR 
INTEGER I 

CHARACTER* 1 0UTLIN(3O 
C OUTLIN AND OUTLINST ARE EQUIVALENCED 

CHARACTER* 1 GCHAR 

C CONSTANT USED THROUGHOUT THE PROGRAM 

CHARACTER* 1 EOTEXT , BLANK, LINEFD 
INTEGER MAXPOS 

COMMON /ALL/ MOREIN, BCOUNT, I, MAXPOS, OUTLIN, 

X EOTEXT, BLANK, LINEFD, GBUFER, GBUF 

DATA EOTEXT, BLANK, LINEFD, MAXPOS / '/', ' ', '&', 31 / 


CALL FIRST 
END 


SUBROUTINE FIRST 
INTEGER K. FILL, BUFPOS 
CHARACTER* 1 CW 
CHARACTER* 1 BUFFER(31) 

INTEGER MOREIN, BCOUNT, I, MAXPOS 

CHARACTER* 1 OUTLINED, GCHAR, EOTEXT, BLANK, LINEFD, 
X GBUFER (80) 

CHARACTER*80 GBUF 

COMMON /ALL/ MOREIN, BCOUNT, I, MAXPOS, OUTLIN, 

X EOTEXT, BLANK, LINEFD, GBUFER, GBUF 

BUFPOS r 0 
FILL = 0 
CW = ' ' 
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ID# 


054 

055 

056 

057 

058 

059 

060 
061 
062 

063 

064 

065 

066 

067 

068 

069 

070 

071 

072 

073 

074 

075 

076 

077 

078 

079 

080 
081 
082 

083 

084 

085 

086 

087 

088 

089 

090 

091 

092 
093: 
094 
095: 
096 
097: 
098 
099: 
100 
101 : 
102 
103: 
104: 
105: 
106 : 


10 


MORE IN = 1 

I = 1 
K = 1 

DOWHILE (K .LE. MAXPOS) 

OUTLIN(K) = ' ' 

K = K + 1 

ENDDO 

BCOUNT = 1 
K = 1 

DOWHILE (K .LE. 80) 

GBUFER(K) = 'Z' 

K = K + 1 

ENDDO ■ 

DOWHILE (MOREIN) 

CW = GCHARO 

IF ((CW .EQ. BLANK) .OR. (CW .EQ. LINEFD) .OR. 

X (CW .EQ. EOTEXT) ) THEN 

IF (CW .EQ. EOTEXT) THEN 
MOREIN = 0 

ENDIF 

IF ( ( FILL+ 1 +BUFP0S ) .LE. MAXPOS) THEN 
CALL PCHAR (BLANK) 

FILL = FILL + 1 

ELSE 

CALL PCHAR (LINEFD) 

FILL =0 

ENDIF 
K = 1 

DOWHILE (K .LE. BUFPOS) 

CALL P CHAR ( BUFFER ( K ) ) 

K = K + 1 

ENDDO 

FILL = FILL + BUFPOS 
BUFPOS = 0 

ELSE 

IF (BUFPOS .EQ. MAXPOS) THEN 
WRITE(6, 10) 

FORMAT ( ' V***WORD TO LONG***') 
MOREIN = 1 

ELSE 

BUFPOS = BUFPOS + 1 
BUFFER (BUFPOS) = CW 

ENDIF 

ENDIF 

ENDDO 

CALL PCHAR (LINEFD) 

END 
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ID# 


107 

108 

109 

110 
111 
112 

113 

114 

115 

116 

117 

118 

119 

120 
121 
122 

123 

124 

125 

126 


128 

129 

130 

131 

132 

133 

134 

135 

136 

137 

138 

139 

140 
141: 
142 
143: 

144 

145 

146 

147 

148 

149 

150 

151 

152 

153 

154 

155 

156 

157 

158 
159 


CHARACTER* 1 FUNCTION GCHARO 
INTEGER MATCH 
CHARACTER*80 GBUFSTR 

INTEGER MOREIN, BCOUNT, I, MAXP0S 

CHARACTER* 1 OUTLINED, EOTEXT, BLANK, LINEFD, 

X GBUFER (80) 

CHARACTER*80 GBUF 

COMMON /ALL/ MOREIN, BCOUNT, I, MAXPOS, OUTLIN, 

X EOTEXT, BLANK, LINEFD, GBUFER , GBUF 

EQUIVALENCE (GBUFSTR, GBUFER) 

IF ( GBUFER ( 1 ) .EQ. 'Z') THEN 
READ (5, 20) GBUF 
20 FORMAT (A80) 

C 

C MATCH ( CARRAY , C , ARSIZE ) RETURNS 1 IF CHARACTER C IS IN CHARACTER ARRAY 
C CARRAY, RETURNS 0 OTHERWISE. ARSIZE IS THE SIZE OF CARRAY. 

C 

IF (MATCH (GBUF, EOTEXT) .EQ. 0) THEN 
WRITE ( 6 , 30) 

30 FORMAT ( ' V***NO END OF TEXT MARK***') 

GBUFER (2) = EOTEXT 

ELSE 

C GBUFER ( 1 ) = GBUF 

GBUFSTR = GBUF 

ENDIF 

ENDIF 

GCHAR = GBUFER (BCOUNT) 

BCOUNT = BCOUNT + 1 
END 


SUBROUTINE PCHAR (C) 

CHARACTER* 1 C 

CHARACTER* 31 SOUT, OUTLINST 
INTEGER K 

INTEGER MOREIN, BCOUNT, I, MAXPOS 

CHARACTER* 1 OUTLINED, GCHAR, EOTEXT, BLANK, LINEFD, 
X GBUFER (80) 

CHARACTER*80 GBUF 

COMMON /ALL/ MOREIN, BCOUNT, I, MAXPOS, OUTLIN, 

X EOTEXT, BLANK, LINEFD, GBUFER, GBUF 

EQUIVALENCE (OUTLINST, OUTLIN) 

IF (C .EQ. LINEFD) THEN 
SOUT = OUTLINST 
WRITE(6,40) SOUT 
40 FORMAT ( ' ',A31) 

K = 1 
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ID# 


160 

161 

162 

163 

164 

165 

166 

167 

168 

169 

170 

171 

172 

173 

174 

175 

176 

177 

178 

179 

180 
181 
182 

183 

184 

185 

186 

187 

188 

189 

190 

191 

192 

193 

194 

195 


DOWHILE (K .LE. MAXPOS) 

OUTLIN(K) = ' ' 

K = K + 1 

ENDDO 
I = 1 

ELSE 

OUTLIN(I) = C 
1 = 1+1 

ENDIF 

END 

C 

C NOTE: YOU DO NOT NEED TO VERIFY THE FOLLOWING FUNCTION. ITS SOURCE 
C CODE IS INCLUDED JUST FOR COMPLETENESS. 

C 

C MATCH ( CARRAY , C , ARSIZE ) RETURNS 1 IF CHARACTER C IS IN CHARACTER ARRAY 
C CARRAY, RETURNS 0 OTHERWISE. ARSIZE IS THE SIZE OF CARRAY. 

C 

INTEGER FUNCTION MATCH (STRIN, CH) 

CHARACTER* 1 00 STRIN 
CHARACTER* 1 CH 

INTEGER PTR 
CHARACTER* 100 STR 
CHARACTER* 1 SARRAY(IOO) 

EQUIVALENCE ( STR, S ARRAY) 

STR = STRIN 
DO 100 PTR = 1,100 

IF (SARRAY(PTR) .EQ. CH) THEN 
MATCH = 1 
RETURN 

ENDIF 

100 CONTINUE 
MATCH = 0 
RETURN 
END 
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PROGRAM 1 FAULTS 


1. Blank is printed before the first word on the first 
line unless the first word is 30 a characters long. 

1. b Leading blank line printed when first word is 

30 a characters long. 

2. Assumes & (not $) is new-line character. 

3. Assumes line size is 31 (not 30) characters. 

4. First character of input line equal to ' z* results 
in line being ignored. 

5. Does not condense successive break characters. 

6. Spelling mistake in 'word to long' (WTL) . 

7. After WTL message, formatting does not terminate. 

7 . b After WTL condition, message printed once for every 

character of word in excess of 30 . 

8. After WTL condition, program prints output buffer. 

9. If a line is entered without the end-of-text (EOT) 

character, a message is printed, but if the next 
line has EOT, this character is changed to 'z' in 
output . 

9. b After two successive omissions of EOT, the program 
prints 'z' and terminates. 

a Actually 31, due to Error 3. 
b Alternate manifestation of this error. 
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2) VAX: Functional Testing 


ID# 


Specification 

A list Is defined to be an ordered collection of elements which may have elements 
annexed and deleted at either end, but not In the middle. The operations that need to 
be available are ADDFIRST, ADDLAST, DELETEFIRST, DELETELAST, FIRST, 
ISEMPTY, LISTLENGTH, REVERSE and NEWLIST. Each operation Is described In 
detail below. 

The lists are to contain up to a maximum of five (5) elements. If an element Is 
added to the front of a "full" list (one containing five elements already), the element at 
the back of the list Is to be discarded. Elements to be added to the back of a full list 
are discarded. Requests to delete elements from empty lists result In an empty list, and 
requests for the first element of an empty list results In zero (0) being returned. The 
detailed operational descriptions are as below. 

ADDFIRST I 

Returns the list with the integer I as its first element followed by all the elements 
of the list. If the list Is "full” to begin with, Its last element Is lost. 

ADDLAST I 

Returns the list with all of the elements of Its elements followed by the Integer I. If 
the list is full to begin with, the list Is returned (l.e., I Is Ignored). 

DELETEFIRST 

Returns the list containing all but Its first element. If the list Is empty, then an 
empty list is returned. 

DELETELAST 

Returns the list containing all but Its last element. If the list Is empty, then an 
empty list Is returned. 

FIRST 

Returns the first element In the list. If the list Is empty, then It returns zero (0). 
ISEMPTY 

Returns one (1) If the list Is empty, zero (0) otherwise. 

LISTLENGTH 

Returns the number of elements In the list. An empty list has zero (0) elements. 
NEWLIST 

Returns an empty list. 

REVERSE 

Returns the list containing Its original elements In reverse order. 

(End of specification.) 
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ID# 


001 

002 : 

003 

004 

005 

006 

007 

008 

009 

010 
011 
012 

013 

014 

015 

016 

017 

018 

019 

020 
021 
022 

023 

024 

025 

026 

027 

028 

029 

030 

031 

032 

033 

034 

035 

036 


C NOTE THAT YOU DO NOT NEED TO VERIFY THE FUNCTIONS DRIVER, GETARG, 

C CHAREQ, CODE, AND PRINT. THEIR SOURCE CODE IS DESCRIBED AND 
C INCLUDED AT THE END FOR COMPLETENESS. 

C NOTE THAT FORMAT STATEMENTS FOR WRITE STATEMENTS INCLUDE A LEADING 
C AND REQUIRED ' ' FOR CARRIAGE CONTROL 
C 

INTEGER POOL (7), LSTEND 
INTEGER LISTSZ 
C 

COMMON /ALL/ LISTSZ 
C 
C 

LISTSZ = 5 

CALL DRIVER (POOL, LSTEND) 

STOP 

END 

C 

Vs 

FUNCTION ADFRST (POOL, LSTEND, I) 

INTEGER ADFRST 
INTEGER POOL ( 7 ) , LSTEND, I 
INTEGER LISTSZ 
COMMON /ALL/ LISTSZ 
C 

INTEGER A 
C 

IF (LSTEND .GT. LISTSZ) THEN 
LSTEND s LISTSZ - 1 

ENDIF 

LSTEND = LSTEND + 1 
A = LSTEND 
DOWHILE (A .GE. 1) 

POOL(A+1 ) = POOL (A) 

A = A - 1 

END DO 
C 


037 

038 
039 


040: 
041: C 
042: C 
043: 


044 

045 


046 

047 

048 

049 


050 

051 


POOL ( 1 ) = I 
ADFRST = LSTEND 
RETURN 
END 


FUNCTION ADLAST (POOL, LSTEND, I) 

INTEGER ADLAST 

INTEGER POOL (7), LSTEND, I 

INTEGER LISTSZ 

COMMON /ALL/ LISTSZ 

IF (LSTEND .LE. LISTSZ) THEN 
LSTEND = LSTEND + 1 
POOL (LSTEND) = I 


052: ENDIF 

053: ADLAST = LSTEND 
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ID# 


054 

055 

056 

057 

058 

059 

060 
061 
062 

063 

064 

065 

066 

067 

068 

069 

070 

071 

072 

073 

074 

075 

076 

077 
073 

079 

080 
08 1 
082 

083 

084 

085 

086 

087 

088 

089 

090 

091 

092 

093 

094 

095 

096 

097 

098 

099 

100 
101 
102 


C 

C 


C 

c 


c 

c 


c 

c 


103 

104 

105 

106 


RETURN 

END 


FUNCTION DELFST (POOL, LSTEND) 
INTEGER DELFST 
INTEGER P00L(7) , LSTEND 
INTEGER LISTSZ 
COMMON /ALL/ LISTSZ 

INTEGER A 

IF (LSTEND .GT. 1) THEN 
A = 1 

LSTEND = LSTEND - 1 
DOWHILE (A .LE. LSTEND) 

POOL(A) = P00L(A+1) 
A = A + 1 

ENDDO 

END IF 

DELFST = LSTEND 

RETURN 

END 


FUNCTION DELLST (LSTEND) 
INTEGER DELLST 
INTEGER LSTEND 

IF (LSTEND .GE. 1) THEN 

LSTEND = LSTEND - 1 

END IF 

DELLST = LSTEND 

RETURN 

END 


FUNCTION FIRST (POOL, LSTEND) 

INTEGER FIRST 

INTEGER POOL (7), LSTEND 


IF (LSTEND .LE. 1) THEN 
FIRST = 0 

ELSE 

FIRST = POOL(1) 


END IF 

RETURN 

END 


C 


FUNCTION EMPTY (LSTEND) 
INTEGER EMPTY 
INTEGER LSTEND 
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107: IF (LSTEND '.LE. 1) THEN 

108: EMPTY = 1 

109: ELSE 

110: EMPTY = 0 

111: END IF 

112: RETURN 

113: END 

114: C 

115: C 

116: FUNCTION LSTLEN (LSTEND) 

117: INTEGER LSTLEN 

118: INTEGER LSTEND 

119: C 

120: LSTLEN = LSTEND - 1 

121: RETURN 

122: END 

123: C 

124: C 

125: FUNCTION NEWLST (LSTEND) 

126: INTEGER NEWLST 

127: INTEGER LSTEND 

128: C 

129: NEWLST = 0 

130: RETURN 

131: END 

132: C 
133: C 

134: SUBROUTINE REVERS (POOL, LSTEND) 

135: INTEGER POOL(7), LSTEND 

136: C 

137: INTEGER I, N 

138: C 

139: N = LSTEND 

140: I = 1 

141: DOWHILE (I .LE. N) 

142: POOL (I) = POOL(N) 

143: N = N - 1 

144: 1=1+1 

145: ENDDO 

146: RETURN 

147: END 

148: C 

149: C 

150: C NOTE: YOU DO NOT NEED TO VERIFY THE FOLLOWING PROCEDURES. THEIR SOURCE 
151: C CODE IS INCLUDED JUST FOR COMPLETENESS. 

152: C 

153: C DRIVER ACCEPTS THE COMMANDS, CALLS THE APPROPRIATE ROUTINES, AND DISPLAYS 
154: C THE RESULTS. 

155: C 

156: SUBROUTINE DRIVER (POOL, LSTEND) 

157: INTEGER POOL (7), LSTEND 

158: C 

159: INTEGER ADFRST , ADLAST , DELFST , DELLST , FIRST , EMPTY , LSTLEN , NEWLST 


A- 11 


2) VAX: Reading Functional Structural 


ID* 


160 

161 

162 

163 

164 

165 

166 

167 

168 

169 

170 

171 

172 

173 

174 

175 

176 

177 

178 

179 

180 
181 
182 

183 

184 

185 

186 

187 

188 
189 

190 

191 

192 

193 

194 

195 

196 

197 

198 

199 

200 
201 
202 

203 

204 

205 

206 

207 

208 

209 

210 
211 
212 


X 


c 

c 


130 

132 

134 

136 


100 


110 


120 


INTEGER ARG, GETARG 
LOGICAL* 1 CODE 
LOGICAL* 1 CMD(80) 

LOGICAL* 1 CMD1 (8 ) ,CMD2(7) ,CMD3( 1 O ,CMD4( 10),CMD5(7) ,CMD6(5), 
CMD7( 1 0 ) , CMD8 ( 7 ) , CMD9 ( 7 ) 

DATA CMD1 /'A','D','D','F','I','R','S','T'/ 

DATA CMD2 /'A ','D ','D ','L ','A ','S ','T'/ 

DATA CMD3 /'D','E','L','E','T','E','FVl','R','S','T'/ 

DATA CMD4 / 'D ','E ','L', 'E ','T', 'E', 'L ','A', 'S ','T'/ 

DATA CMD5 /'I ','S', 'E', 'M','P', 'T ','T'/ 

DATA CMD6 /'F ','1', 'R ','S','T'/ 

DATA CMD7 / 'L' , 'I' , 'S ', 'T', 'L ' , 'E ', 'N', 'G', 'T', 'H'/ 

DATA CMD8 / 'R ' , 'E ' , 'V ' , 'E ' , 'R ' , 'S ' , 'E '/ 

DATA CMD9 /'N'/E'.T/L'.'l'.'S'/T'/ 


LSTEND = NEWLST ( LSTEND ) 

EXECNT = 1 
D0WHILE (1 .EQ. 1) 

READ (5, 1 30,END=999) CMD 
FORMAT (80A1) 

WRITE (6, 132) EXECNT 
FORMAT ( ' ','< INPUT M3,' : >') 

WRITEC6, 134) CMD 
FORMAT ( ' '< ' ,60A1 , '> ') 

WRITE(6,136) 

FORMAT ( ' ','< OUTPUT : >') 

EXECNT s EXECNT +■ 1 
IF ( CODE ( CMD , 80 , CMD 1,8)) THEN 
ARG = GETARG (CMD ,80) 

LSTEND = ADFRST( POOL, LSTEND, ARG) 
ELSE IF (C0DE(CMD ,80,CMD2 ,7) ) THEN 
ARG = GET ARG (CMD, 80) 

LSTEND = ADL AST (POOL, LSTEND, ARG) 
ELSEIF ( CODE ( CMD , 80 , CMD3 ,11)) THEN 
LSTEND = DELFST( POOL, LSTEND) 
ELSEIF (CODE(CMD,80,CMD4, 10) ) THEN 
LSTEND = DELLST( LSTEND) 

ELSEIF (CODE(CMD,30,CMD5,7) ) THEN 
ITMP = EMPTY (LSTEND) 

WRITE(6, 100) ITMP 
FORMAT ( ' ',' VALUE IS ',110) 

ELSEIF (C0DE(CMD,80,CMD6,5) ) THEN 
ITMP = FIRST (POOL, LSTEND) 
WRITE(6, 110) ITMP 
FORMAT ( ' ',' VALUE IS ',110) 

ELSEIF (C0DE(CMD,30,CMD7, 10) ) THEN 
ITMP = LSTLEN( LSTEND) 

WRITE(o, 120) ITMP 
FORMAT ( ' ',' VALUE IS ',110) 

ELSEIF ( CODE ( CMD , 30 , CMD8 , 7 ) ) THEN 
CALL REVERS( POOL, LSTEND) 

ELSEIF ( CODE ( CMD , 80 , CMD9 , 7 ) ) THEN 
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213 

LSTEND = NEWLST ( LSTEND ) 

214 

ELSE 

215 

WRITE (6, 2 10) CMD 

216 

210 FORMAT ( ' UNKNOWN COMMAND: ',50A1,'>') 

217 

END IF 

218 

C 

219 

C PRINT PRINTS THE LIST 

220 

C 

221 

CALL PRINT (POOL, LSTEND) 

222 

ENDDO 

223 

999 CONTINUE 

224 

RETURN 

225 

END 

226 

C 

227 

C CODE ( CA 1 , LEN 1 , CA2 , LEN2 ) RETURNS TRUE IF THE FIRST STRING IN CHARACTER 

228 

C ARRAY CA1 IS EQUIVALENT TO THE FIRST STRING IN CA2. IT RETURNS FALSE 

229 

C OTHERWISE. 

230 

C 

231 

FUNCTION CODE (CA1 ,LEN1 ,CA2 ,LEN2) 

232 

LOGICAL* 1 CODE, CA1(120), CA2(120) 

233 

INTEGER LEN1 , LEN2 

234 

C 

235 

INTEGER 11, 12 

236 

LOGICAL* 1 CHAREQ 

237 

11 = 1 

238 

12 = 1 

239 

DOWHILE ((11 .LE. LEN1). AND. CHAREQ(CA1 (11 ) , ' ') ) 

240 

11 s 11 + 1 

241 

ENDDO 

242 

DOWHILE ((12 .LE. LEN2) .AND. CHAREQ (CA2( 12) , ' ') ) 

243 

12 s 12 + 1 

244 

ENDDO 

245 

C 

246 

DOWHILE ( (CHAREQ(CA1 (11 ) ,CA2(I2) ) ) .AND. 

247 

X ( .NOT. (CHAREQ(CA1 (11 ) , * ') ) ) .AND . ( .NOT. ( CHAREQ (CA2( 12) , ' '))) 

248 

X .AND. (11 .LE. LEND. AND. (12 .LE. LEN2) ) 

249 

11 = 11 + 1 

250 

12 = 12 + 1 

251 

ENDDO 

252 

IF ( ( (LEN2 .LE. LEN1). AND. (12 .GT. LEN2) ) .OR. ; 

253 

X ( (LEN2 .GT. LEND. AND. (11 .GT. LEND) ) THEN 

254 

CODE = 1 

255 

ELSE 

256 

CODE = 0 

257 

END IF 

258 

RETURN 

259 

END 

260 

C 

261 

C CHAREQ(C1 ,C2) IS CHARACTER EQUIVALENCE; RETURNS TRUE IF CHARACTER Cl 

262 

C EQUALS CHARACTER C2, RETURNS FALSE OTHERWISE 

263 

C 

264 

FUNCTION CHAREQ (C1,C2) 

265 

LOGICAL* 1 CHAREQ, Cl, C2 
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ID# 


266 : 

267 

268 
269: 
270: 

271 

272 

273 

274 

275 

276 

277 

278 

279 

280 
281 
282 
283 

284 

285 

286 

287 

288 

289 

290 

291 

292 

293 

294 

295 

296 

297 

298 

299 

300 

301 

302 

303 

304 

305 

306 

307 

308 

309 

310 

311 

312 

313 

314 


C 

IF (Cl .EQ. C2) THEN 
CHAREQ = 1 

ELSE 

CHAREQ = 0 

ENDIF 

RETURN 

END 

C 

C GETARG RETURNS THE SECOND STRING IN CHARACTER ARRAY CA CONVERTED TO AN 
C INTEGER. 

C 

FUNCTION GETARG (CA, LEN) 

INTEGER GETARG, LEN 
LOGICAL* 1 CA( 120) 

C 

INTEGER I, SUM, NEGSUM 
LOGICAL* 1 CHAREQ 
I = 1 

C STRIP LEADING BLANKS 

DOWHILE ((I .LE. LEN). AND. CHAREQ(CA(I) , ' ') ) 

1 = 1+1 

ENDDO 

C SKIP OVER COMMAND NAME 

DOWHILE ((I .LE. LEN) .AND. ( .NOT. (CHAREQ(CA(I) , ' '))) ) 

1 = 1+1 

ENDDO 

C SKIP BLANKS BETWEEN COMMAND AND ARGUMENT 

DOWHILE ((I .LE. LEN). AND. CHAREQ(CA(I) , ' ') ) 

1 = 1+1 

ENDDO 

C CALCULATE ARGUMET 
SUM = 0 
NEGSUM = 0 

DOWHILE ((I .LE. LEN) . AND. ( .NOT. (CHAREQ(CA(I) , ' '))) ) 

IF (CHAREQ(CA(I) ,'-')) THEN 
NEGSUM = 1 

ELSE 

SUM = SUM * 10 + CA(I) - 48 

ENDIF 
1 = 1+1 

ENDDO 

IF (NEGSUM .EQ. 0) THEN 
GETARG = SUM 

ELSE 

GETARG = -SUM 

ENDIF 

RETURN 

END 
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1. FIRST returns 0 when the list has one element. 

2. ISEMPTY returns 1 when the list has one element. 

3. DELETEFIRST cannot delete the first element when 
the list has one element. 

4. LISTLENGTH returns 1 less than the actual length of 
the list. 

5. ADDFIRST can add more than five elements to the 
list. 

6. ADDLAST can add more than five elements to the list. 

7. REVERSE does not reverse the list properly. 
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VAX & IBM: Structural Testing 


Specification 

(Note from Rick: a 'file’ Is the same thing as an IBM 'dataset’.) 

The program maintains a database of bibliographic references. It first reads a mas- 
ter file of current references, then reads a file of reference updates, merges the two, and 
produces an updated master file and a cross reference table of keywords. 

The first Input file, the master, contains records of 74 characters with the following 
format: 

column comment 

1-3 each reference has a unique reference key 
4-14 author of publication 
15 - 72 title of publication 
73 - 74 year issued 

.The -key.- should be a three (3) character unique Identifier consisting of letters between 
A-Z. The next Input file, the update file, contains records of 75 characters In length. 
The only difference from a master file record Is that an update record has either an 'A’ 
(capital A meaning add) or a ’R’ (capital R meaning replace) In column 75. Both the 
master and update files are expected to be already sorted alphabetically by reference key 
when read Into the program. Update records with action replace are substituted for the 
matching key record In the master file. Records with action add are added to the mas- 
ter file at the appropriate location so that the file remains sorted on the key field. For 
example, a valid update record to be read would be (Including a numbered line Just for 
reference) 

1 2345678001234507890 12345Q78901 2345678901 2345678901 2345678901 23456789012345 

BITbaker an Introduction to program testing 83A 

The program should produce two pieces of output. It should first print the sorted 
list of records In the updated master file In the same format as the original master file. 
It should then print a keyword cross reference list. All words greater than three charac- 
ters In a publication’s title are keywords. These keywords are listed alphabetically fol- 
lowed by the key fields from the applicable updated master file entries. For example. If 
the updated master file contained two records, 

ABCkennlt Introduction to software testing 82 

EDKJones the realities of software management 81 

then the keywords are introduction, testing, realities, software, and management. The 
cross reference list should look like 

Introduction 

ABC 

management 

DDX 
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realities 

DDX 

software 

ABC 

DDX 

testing 

ABC 

Some possible error conditions tliat could arise and the subsequent actions Include 
the following. The master and update files should be checked for sequence, and If a 
record out of sequence Is found, a message similar to 'key ABC out of sequence' should 
appear and the record should be discarded. If an update record Indicates replace and 
the matching key can not be found, a message similar to 'update key ABC not found’ 
should appear and the update record should be Ignored. If an update record Indicates 
add and a matching key Is found, something like 'key ABC already In file' should 
appear and the record should be Ignored. (End of specification.) 
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ID# 


001 

002 

003 

004 

005 

006 
007 
008: 
009 
010 : 
011 
012 

013 

014 

015 

016 

017 

018 

019 

020 
021 
022 

023 

024 

025 

026 

027 

028 

029 

030 

031 

032 

033 

034 

035 

036 

037 

038 

039 

040 

041 

042 

043 

044 

045 

046 

047 

048 

049 

050 

051 

052 
053 


C NOTE THAT YOU DO NOT NEED TO VERIFY THE ROUTINES DRIVER, STREQ, WORDEQ, 
C NXTSTR , ARRCPY, CHARPT, BEFORE, CHAREQ, AND WRDBEF. THEIR SOURCE 
C CODE IS DESCRIBED AND INCLUDED AT THE END FOR COMPLETENESS. 

C NOTE THAT FORMAT STATEMENTS FOR WRITE STATEMENTS INCLUDE A LEADING 
C AND REQUIRED ' ' FOR CARRIAGE CONTROL 

C THE SFORT LANGUAGE CONSTRUCT '.IF (EXPRESSION)' BEGINS A BLOCKED 
C IF-THEN [-ELSE] STATEMENT, AND IT IS EQUIVALENT TO THE F77 
C 'IF (EXPRESSION) THEN'. 

C 

CALL DRIVER 
STOP 
END 
C 
C 

SUBROUTINE MAINSB 
C 

LOGICAL* 1 U$KEY ( 3 ) , U$ AUTH ( 1 1 ) , U$TITL ( 58 ) , U$ YEAR ( 2 ) , U$ ACTN ( 1 ) 
LOGICAL* 1 M$KEY( 3) , M$AUTH (11), M$TITL ( 58 ) , M$ YEAR ( 2 ) 

LOGICAL* 1 ZZZ(3), LASTUK ( 3 ) , LASTMK(3) 

LOGICAL* 1 STREQ, CHAREQ, BEFORE, CHARPT 
INTEGER I 
C 

LOGICAL* 1 WORD (500, 12) , REFKEY( 1000,3) 

INTEGER NUMWDS, NUMREF, PTR(500), NEXT (1000) 

COMMON /WORDS/ WORD, REFKEY , NUMWDS, NUMREF, PTR, NEXT 
C 

WRITE (6, 290) 

290 FORMAT (' ',' UPDATED LIST OF MASTER ENTRIES') 

DO 300 I = 1, 3 

LASTMK ( I ) = CHARPT ( ' ') 

LASTUK ( I ) = CHARPT (' ') 

ZZZ(I) = CHARPT ( 'Z ') 

300 CONTINUE 
C 

NUMWDS = 0 
NUMREF = 0 

CALL GETNM(M$KEY , M$AUTH,M$TITL,M$ YEAR , LASTMK) 

CALL GETNUP ( U$KEY , U$ AUTH , U$TITL , U$ YEAR , U$ ACTN , LASTUK ) 

C 

DOWHILE ( ( .NOT. (STREQ(M$KEY,ZZZ,3) ) ) .OR. 

X ( .NOT. (STREQ(U$KEY,ZZZ , 3) ) ) ) 

.IF (STREQ(U$KEY,M$KEY,3) ) 

.IF ( .NOT. ( CHAREQ (U$ ACTN ( 1 ) , 'R ') ) ) 

WRITE (6, 100) U$KEY 

100 FORMAT (' ' , 'KEY ' , 3A1 , ' IS ALREADY IN FILE') 

ENDIF 

CALL OUTPUT ( U$KEY , U$ AUTH , U$TITL , U$ YEAR ) 

CALL DICTUP ( U$KEY , U$TITL , 58 ) 

CALL GETNM(M$KEY , M$ AUTH , M$TITL , M$YEAR , LASTMK ) 

CALL GETNUP ( U$KEY , U$ AUTH , U$TITL , U$ YEAR , U$ ACTN , LASTUK ) 

ENDIF 


C 


.IF (BEFORE(M$KEY,3,U$KEY,3) ) 
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ID# 


054 

055 

056 

057 

058 

059 

060 
061 : 
062 

063 

064 

065 

066 
067 
068 

069 

070 

071 

072 

073 

074 

075 

076 

077 

078 

079 

080 
081 
082 

083 

084 

085 

086 

087 

088 

089 

090 

091 

092 

093 

094 

095 

096 

097 

098 

099 

100 
101 
102 

103 

104 

105 

106 


CALL OUTPUT (M$KEY ,M$ AUTH , M$TITL , M$YEAR ) 

CALL DICTUP (M$KEY ,M$TITL , 58 ) 

CALL GETNM (M$KEY, M$ AUTH , M$TITL , M$ YE AR , LASTMK ) 

END IF 
C 

.IF ( BEFORE (U$KEY,3,M$KEY,3)) 

.IF (CHAREQ(U$ACTN(1), 'R')) 

WRITE(6,110) U$KEY 

110 FORMAT ( ' ', 'UPDATE KEY ',3A1,' NOT FOUND') 

END IF 

CALL OUTPUT ( U$KEY , U$ AUTH , U$TITL , U$ YEAR ) 

CALL DICTUP (U$KEY,U$TITL, 58) 

CALL GETNUP (U$KEY , U$AUTH , U$TITL , U$ YEAR , U$ ACTN , LASTUK ) 
ENDIF 
ENDDO 
C 

CALL SRTWDS 
CALL PRTWDS 
RETURN 
END 
C 
C 

SUBROUTINE GETNM ( KEY , AUTH , TITL , YEAR , LASTMK ) 

LOGICAL* 1 KEY( 3) ,AUTH(1 1 ) ,TITL(58) , YEAR (2) ,LASTMK(3) 

C 

LOGICAL* 1 SEQ, INLINE (80) 

LOGICAL* 1 BEFORE, CHARPT, CHAREQ 
LOGICAL* 1 GO$M, GO$U 
COMMON /DRIV/ GO$M, GO$U 
C 

SEQ = 1 
DOWHILE (SEQ) 

.IF (GO$M) 

C 

C READ FROM THE MASTER FILE 
C 

READ(10,200,END=299) INLINE 
ELSE 
C 

C SEE REMARK ABOUT THE CHARACTER 'V LATER IN THE ROUTINE. 

C 

INLINE ( 1 ) = CHARPT ( '%') 

ENDIF 

200 FORMAT (80A1) 

DO 210 I = 1, 3 

KEY(I) = INLINE(I) 

210 CONTINUE 

DO 220 I = 1, 11 

AUTH ( I ) = INLINE (3+1) 

220 CONTINUE 

DO 230 I = 1, 58 

TITL(I) = INLINE (14+1) 

230 CONTINUE 
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ID# 


107 

108: 

109 

110 
111 
112 

113 

114 

115 

116 

117 

118 

119 

120 
121 
122 

123 

124 

125 

126 

127 

128 

129 

130 

131 

132 

133 

134 

135 

136 

137 

138 

139 

140 

141 

142 

143 

144 

145 

146 

147 

148 

149 

150 

151 

152 

153 

154 

155 

156 

157 

158 
159 


DO 240 I = 1, 2 

YEAR(I) = INLINE (72+1) 

240 CONTINUE 

C 

C A METHOD OF SPECIFYING END-OF-FILE IN A FILE IS TO PUT THE CHARACTER 
C AS THE FIRST CHARACTER ON A LINE. THE DRIVER USES THIS FOR MULTIPLE 
C SETS OF INPUT CASES. 

C 

.IF ((. NOT. (CHAREQ(KEY(1 ),'%'))) .AND. 

X (BEFORE(KEY,3,LASTMK,3)) ) 

WRITE (6, 250) KEY 

250 FORMAT (' ' , 'KEY ',3A1,' OUT OF SEQUENCE') 

ELSE 

CALL ARRCP Y ( KEY , LASTMK , 3 ) 

SEQ = 0 
END IF • 

.IF (CHAREQ(KEY(1) t '%')) 

SEQ = 0 

DO 270 I = 1, 3 

KEY(I) = CHARPT('Z') 

270 CONTINUE 

ENDIF 
ENDDO 
RETURN 

299 CONTINUE 
GO$M = 0 
DO 260 I = 1, 3 

KEY(I) r CHARPT('Z') 

260 CONTINUE 
RETURN 
END 
C 
C 

SUBROUTINE GETNUP ( KEY , AUTH , TITL , YE AR , ACTN , LASTUK ) 

LOGICAL* 1 KEY ( 3 ) , AUTH (11), TITL ( 5 8 ) , YE AR ( 2 ) , ACTN ( 1 ) , LASTUK ( 3 ) 

C 

LOGICAL* 1 SEQ, INLINE (80) 

LOGICAL* 1 BEFORE, CHARPT, CHAREQ 
LOGICAL* 1 GO$M, GO$U 
COMMON /DRIV/ GO$M, GO$U 
C 

SEQ = 1 
DOWHILE (SEQ) 

.IF (GO$U ) 

C 

C READ FROM THE UPDATES FILE 
C 

READ( 1 1 ,200,END=299) INLINE 
ELSE 
C 

C SEE REMARK ABOUT THE CHARACTER LATER IN THE ROUTINE. 

C 

INLINE ( 1 ) = CHARPT ( '%') 
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160 

161 : 

162 

163 

164 

165 

166 

167 

168 

169 

170 

171 

172 

173 

174 

175 

176 

177 

178 

179 

180 
181 
182 

183 

184 

185 

186 

187 

188 

189 

190 

191 

192 

193 

194 

195 

196 

197 

198 

199 

200 
201 
202 

203 

204 

205 

206 

207 

208 

209 

210 
211 
212 


ID# 


ENDIF 

200 FORMAT (80A1) 

DO 210 I = 1, 3 

KEY(I) = INLINE (I) 

210 CONTINUE 

DO 220 I = 1, 11 

AUTH(I) = INLINE (3+D 
220 CONTINUE 

DO 230 I = 1, 58 

TITL(I) = INLINE (14+1) 

230 CONTINUE 

DO 240 I = 1, 2 

YEAR(I) = INLINE (72+1). 

240 CONTINUE 

ACTN(1) = INLINE (75) 

C 

C A METHOD OF SPECIFYING END-OF-FILE IN A FILE IS TO PUT THE CHARACTER 
C AS THE FIRST CHARACTER ON A LINE. THE DRIVER USES THIS FOR MULTIPLE 
C SETS OF INPUT CASES. 

C 

.IF ((. NOT. (CHAREQ(KEY(1 ), '%'))) .AND. 

X (BEFORE (KEY, 3 ,LASTUK, 3) ) ) 

WRITE (6, 250) KEY 

250 FORMAT ( ' ' , 'KEY ',3A1,' OUT OF SEQUENCE') 

ELSE 

CALL ARRCPY ( KEY , LASTUK , 3 ) 

SEQ = 0 
ENDIF 

.IF (CHAREQ(KEY( 1 ),'%')) 

SEQ = 0 

DO 270 I = 1, 3 

KEY(I) = CHARPT('Z') 

270 CONTINUE 

ENDIF 
ENDDO 
RETURN 

299 CONTINUE 
GO$U = 0 
DO 260 I = 1, 3 

KEY(I) = CHARPT('Z') 

260 CONTINUE 
RETURN 
END 
C 
C 

■ SUBROUTINE OUTPUT (KEY, AUTH,TITL, YEAR ) 

LOGICAL* 1 KEY ( 3 ) > AUTH(11), TITL(58), YEAR (2) 

C 

WRITE (6, 200) KEY, AUTH, TITL, YEAR 
200 FORMAT ( ' ',3A1 , 1 1A1 ,58A1 ,2A1 ) 

RETURN 

END 

C 
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213: 

214: 

215: 

216: 

217: 

218: 

219: 

220 : 

221 : 

222 : 

223: 

224: 

225: 

226: 

227: 

228: 

229 : 

230: 

231: 

232 : 

233: 

234: 

235: 

236: 

237: 

238: 

239: 

240: 

241: 

242: 

243: 

244: 

245: 

246: 

247: 

248: 

249: 

250: 

251: 

252: 

253: 

254: 

255: 

256 : 

257: 

258 : 

259: 

260 : 

261 : 

262: 

263: 

264: 

265: 


ID# 


C 

SUBROUTINE PRTWDS 
C 

LOGICAL* 1 WORD (500, 12) , REFKEY( 1000, 3) 

INTEGER NUMWDS , NUMREF, PTR(500), NEXT (1000) 

COMMON /WORDS/ WORD, REFKEY, NUMWDS, NUMREF, PTR, NEXT 
C 

C THE ABOVE GROUP OF DATA STRUCTURES SIMULATES A LINKED LIST. 

C WORD ( I , J ) IS A KEYWORD — J RANGING FROM 1 TO 12 
C REFKEY ( PTR ( I ),K),K= 1,3 IS THE FIRST 3 LETTER KEY THAT HAS AS A 
C KEYWORD WORD(I, J) , J=1 ,'12 

C REFKEY(NEXT(PTR(I)),K),K=1,3 IS THE SECOND 3 LETTER KEY THAT HAS 
C AS A KEYWORD WORD (I, J) , J=1 , 12 

C REFKEY (NEXT (NEXT (PTR( I) ) ) ,K) ,K=1 ,3 IS THE THIRD ... ETC. 

C NEXT ( J ) IS EQUAL TO -1 WHEN THERE ARE NO MORE 3 LETTER KEYS FOR 
C THE PARTICULAR KEYWORD 
C 

INTEGER I, J 
LOGICAL* 1 FLAG 
C 

WRITE (6,200) 

200 FORMAT (' KEYWORD REFERENCE LIST') 

DO 210 I = 1, NUMWDS 
FLAG = 1 

WRITE( 6,220) (WORD(I, J) , J=1 , 12) 

220 FORMAT ( ' ',12A1) 

LAST = PTR ( I ) 

DOWHILE (FLAG) 

WRITE(6,230) (REFKEY(LAST,J) ,J=1 ,3) 

230 FORMAT ( ' ',3A1) 

LAST = NEXT (LAST) 

.IF (LAST .EQ. -1) 

FLAG = 0 
ENDIF 
ENDDO 

210 CONTINUE 
RETURN 
END 
C 

c 

SUBROUTINE DICTUP ( KEY , STR , STRLEN ) 

LOGICAL* 1 KEY ( 3 ) > STR (120) 

INTEGER STRLEN 
C 

LOGICAL* 1 WDLEFT , FLAG, OKLEN, NEXTWD(120), WORDEQ 
INTEGER LPTR , NXTSTR, LEN, LAB, I, K 
C 

LOGICAL* 1 WORD (500, 12) , REFKEY ( 1000,3) 

INTEGER NUMWDS, NUMREF, PTR (500), NEXT (1000) 

COMMON /WORDS/ WORD, ' REFKEY, NUMWDS, NUMREF, PTR, NEXT 
C 

C THE ABOVE GROUP OF DATA STRUCTURES SIMULATES A LINKED LIST. 

C WORD ( I , J ) IS A KEYWORD — J RANGING FROM 1 TO 12 
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ID# 


266 

267 

268 

269 

270 

271 

272 

273 

274 


C REFKEY(PTR(I) , K) ,K= 1 , 3 IS THE FIRST 3 LETTER KEY THAT HAS AS A 
C KEYWORD W0RD(I, J) ,J=1 , 12 

C REFKEY(NEXT(PTR(I)),K),K=1,3 IS THE SECOND 3 LETTER KEY THAT HAS 
C AS A KEYWORD WORD(I , J) , J=1 , 12 

C REFKEY( NEXT ( NEXT (PTR(I) ) ) ,K) ,K=1 ,3 IS THE THIRD ... ETC. 

C NEXT ( J ) IS EQUAL TO -1 WHEN THERE ARE NO MORE 3 LETTER KEYS FOR 
C THE PARTICULAR KEYWORD 
C 

WDLEFT = 1 


275 

276 

277 

278 

279 

280 
281 
282 

283 

284 

285 

286 

287 

288 

289 

290 

291 

292 

293 

294 

295 

296 

297 

298 

299 

300 
301 
302 

303 

304 

305 

306 

307 

308 

309 

310 

311 

312 

313 

314 

315 

316 

317 

318 


C 


C 


C 


300 


310 


320 


LPTR = 1 

DOWHILE (WDLEFT) 

FLAG = 1 
OKLEN = 1 

LEN = NXTSTR (STR,STRLEN, LPTR, NEXTWD, 120) 
.IF (LEN .EQ. 0) 

WDLEFT = 0 
ENDIF 

.IF (LEN .LE. 2) 

OKLEN = 0 
ENDIF 

.IF (OKLEN) 

I = 1 

DOWHILE ((I .LE. NUMWDS ) . AND . FLAG ) 
.IF (WORDEQ( NEXTWD, I)) 

LAB = I 
FLAG = 0 
ENDIF 
1 = 1+1 
ENDDO 
.IF (FLAG) 

NUMWDS = NUMWDS + 1 
NUMREF = NUMREF + 1 
DO 300 K = 1, 12 

WORD ( NUMWDS, K) = NEXTWD (K) 

. CONTINUE 

PTR( NUMWDS) = NUMREF 
DO 310 K = 1, 3 

REFKEY( NUMREF, K) = KEY(K) 
CONTINUE 

NEXT (NUMREF) = -1 
ELSE 

NUMREF = NUMREF + 1 
DO 320 K = 1, 3 

REFKEY ( NUMREF, K) = KEY(K) 
CONTINUE 

NEXT (NUMREF) = PTR(LAB) 

PTR(LAB) = NUMREF 
ENDIF 
ENDIF 
ENDDO 
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ID# 


319: 

C 




320: 


RETURN 



321: 


END 



322: 

C 




323: 

C 




324: 


SUBROUTINE SRTWDS 



325: 

C 




326: 


LOGICAL* 1 WORD(500, 12) , REFKEY( 1000,3) 



327: 


INTEGER NUMWDS , NUMREF, PTR(500), NEXT (1000) 



328: 


COMMON /WORDS/ WORD, REFKEY , NUMWDS, NUMREF, PTE 

:, NEXT 


329: 

C 




330: 

C THE ABOVE GROUP OF DATA STRUCTURES SIMULATES A LINKED LIST. 


331: 

C WORD(I , J ) IS A KEYWORD — J RANGING FROM 1 TO 12 



332: 

C REFKEY(PTR(I),K),K=1,3 IS THE FIRST 3 LETTER KEY THAT HAS AS 

A 

333: 

C 

KEYWORD WORD(I,J) , J=1 , 12 



334: 

C REFKEY(NEXT(PTR(I)),K),K=1,3 IS THE SECOND 3 LETTER 

KEY THAT 

HAS 

335: 

C 

AS A KEYWORD WORDU, J) , J=1 , 12 



336: 

C REFKEY(NEXT(NEXT(PTR(I))),K),K=1,3 IS THE THIRD ... 

ETC. 


337: 

C NEXT ( J ) IS EQUAL TO- -1 WHEN THERE ARE NO MORE 3 LETTER KEYS 

FOR 

338: 

C 

THE PARTICULAR KEYWORD 



339: 

C 




340: 


INTEGER I, J, K, LAB, LOWERB, UPPERB 



341: 


LOGICAL* 1 WRDBEF , NEXTWD(12) 



342: 

C 




343: 


UPPERB = NUMWDS - 1 



344: 


DO 400 I = 1, UPPERB 



345: 


LOWERB =1+1 



346: 


DO 410 J = LOWERB, NUMWDS 



347: 


.IF (WRDBEF(J,I)) 



348: 


DO 300 K = 1, 12 



349: 


NEXTWD(K) = WORDU, K) 



350: 

300 

CONTINUE 



351: 


LAB = PTR(I) 



352: 


DO 310 K = 1, 12 



353: 


WORDU, K) = WORDCJ, K) 



354: 

310 

CONTINUE 



355: 


PTR(I) = PTR(J) 



356: 


DO 320 K = 1, 12 



357: 


WORD ( J , K ) = NEXTWD(K) 



358: 

320 

CONTINUE 



359: 


PTR(J) = LAB 



360: 


ENDIF 



361: 

410 

CONTINUE 



362: 

400 

CONTINUE 



363: 

C 




364: 


RETURN 



365: 


END 



366: 

C 




367: 

C 




368: 

C NOTE: YOU DO NOT NEED TO VERIFY THE FOLLOWING PROCEDURES. THEIR 

369: 

C 

CODE IS INCLUDED JUST FOR COMPLETENESS. 



370: 

C 




371: 

C DRIVER CONTINUES TO CALL THE MAIN ROUTINE UNTIL END- 

■OF-FILE 

HAS 
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ID# 


372 

373 

374 

375 

376 

377 

378 

379 

380 
381 

382 

383 

384 

385 

386 

387 

388 

389 

390 

391 

392 

393 

394 

395 

396 

397 

398 

399 

400 

401 

402 

403 

404 

405 

406 

407 

408 

409 

410 

411 

412 

413 

414 

415 

416 

417 

418 

419 

420 

421 

422 

423 

424 


C REACHED ON BOTH THE MASTER AND UPDATES DATASETS. 

C 

SUBROUTINE DRIVER 
C 

INTEGER EXECNT 
LOGICAL*1 GO$M, GO$U 
COMMON /DR IV/ GO$M, GO$U 
C 

EXECNT = 1 
GO$M = 1 
GO$U = 1 

DOWHILE ( (GO$M) .OR. (GO$U) ) 

WRITE (6, 90) EXECNT 

90 FORMAT ( ' ','< PROGRAM EXECUTION ',13,' : >') 

CALL MAINSB 
EXECNT = EXECNT + 1 
ENDDO 
RETURN 
END . 

C 

C WRDBEF(P1,P2) RETURNS 1 IF THE KEYWORD IN WORD(P1 ,J) ,J=1 , 12 COMES 
C ALPHABETICALLY BEFORE THE KEYWORD IN WORD(P2, J) , J=1 , 12. 

C IT RETURNS 0 OTHERWISE. 

C 

FUNCTION WRDBEF(PTR1 ,PTR2) 

LOGICAL* 1 WRDBEF 
INTEGER PTR1 , PTR2 
C 

LOGICAL* 1 WORD (500, 12) , REFKEY( 1000,3) 

INTEGER NUMWDS, NUMREF, PTR(500), NEXT (1000) 

COMMON /WORDS/ WORD, REFKEY , NUMWDS, NUMREF, PTR, NEXT 
LOGICAL* 1 BEFORE, W0RD1(12), W0RD2(12) 

INTEGER I 
C 

DO 500 I = 1, 12 

word id) = Word ( ptr i ,i) 

W0RD2(I) = W0RD(PTR2,I) 

500 CONTINUE 

.IF ( BEFORE ( WORD 1 , 12,W0RD2, 12) ) 

WRDBEF = 1 
ELSE 

WRDBEF = 0 
END IF 
RETURN 
END 
C 

C CHAREQ(C1 ,C2) IS CHARACTER EQUIVALENCE; RETURNS TRUE IF CHARACTER Cl 
C EQUALS CHARACTER C2, RETURNS FALSE OTHERWISE 
C 

FUNCTION CHAREQ (C1,C2) 

LOGICAL* 1 CHAREQ, Cl , C2 
C 

• IF (Cl. .EQ. C2) 
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ID# 


425 

426 

427 

428 

429 

430 

431 

432 

433 

434 

435 

436 

437 

438 

439 

440 

441 

442 

443 

444 

445 

446 

447 

448 

449 

450 

451 

452 

453 

454 

455 

456 

457 

458 

459 

460 

461 

462 

463 

464 

465 

466 

467 

468 

469 

470 

471 

472 

473 

474 

475 

476 

477 


CHAREQ = 1 
ELSE 

CHAREQ = 0 
ENDIF 
RETURN 
END 
C 

C BEFORE ( S 1,L1,S 2, L2) RETURNS 1 IF THE STRING IN CHAR ARRAY SI COMES 
C ALPHABETICALLY BEFORE THE STRING IN CHAR ARRAY S2. SI AND S2 ARE 
C OF SIZES LI AND L2. IT RETURNS 0 OTHERWISE (INCLUDING WHEN THE 
C STRINGS ARE EXACTLY THE SAME). 

C 

FUNCTION BEFORE (STR1 ,LEN1 ,STR2,LEN2 ) 

LOGICAL* 1 BEFORE, STR1 (120) , STR2(120) 

INTEGER LEN1 , LEN2 
C 

INTEGER PTR 
LOGICAL* 1 TIE 
PTR = 1 
TIE = 1 

DOWHILE ( TIE .AND. (PTR.LE.LEN1 ) . AND. (PTR.LE.LEN2) ) 

.IF (STRI(PTR) .LT. STR2(PTR)) 

BEFORE = 1 
TIE = 0 
ELSE 

.IF (STR1 (PTR) .GT. STR2(PTR)) 

BEFORE = 0 
TIE s 0 
ENDIF 
ENDIF 

PTR = PTR + 1 
ENDDO 
.IF (TIE) 

.IF ((PTR .GT. LEND. AND. (PTR .LE. LEN2) ) 

BEFORE = 1 
ELSE 

BEFORE = 0 
ENDIF 
ENDIF 
RETURN 
END 
C 

C CHARPT ('C') IS A CHARACTER ASSIGNMENT FUNCTION; IT RETURNS THE 
C CHARACTER PASSED TO IT AS AN ARGUMENT 
C 

FUNCTION CHARPT(C) 

LOGICAL* 1 CHARPT, C 
C 

CHARPT = C 
RETURN 
END 
C 

C ARRCPY(ARR1,ARR2,ARSIZE) COPIES CHARACTER ARRAY ARR1 TO CHARACTER 
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ID# 


478 

479 

480 

481 

482 

483 

484 

485 

486 

487 

488 

489 

490 

491 

492 

493 

494 

495 

496 

497 

498 

499 
500: 
501 

502 

503 

504 

505 

506 

507 

508 

509 

510 

511 

512 

513 

514 

515 

516 

517 

518 

519 

520 

521 

522 

523 

524 

525 

526 

527 

528 

529 

530 


C ARRAY ARR2. THE ARRAYS ARE OF SIZE ARSIZE 
C 

SUBROUTINE ARRCPY (ARR1 ,ARR2,LEN) 

LOGICAL* 1 ARR1C120), ARR2(120) 

INTEGER LEN 
C 

INTEGER I 

DO 230 I = 1, LEN 

ARR2(I) = ARRI(I) 

230 CONTINUE 
RETURN 
END 
C 

C NXTSTR(STR, LEN, START, NEXTST , NXTLEN ) COPIES THE NEXT STRING IN 
C CHAR ARRAY STR TO CHAR ARRAY NEXTST. THE FORWARD SEARCH FOR THE 
C NEXT STRING BEGINS AT INDEX START. START IS UPDATED TO BE THE INDEX OF THE 
C CHARACTER IMMEDIATELY FOLLOWING THE STRING IN STR. 

C THE SIZE OF STR IS LEN. THE LENGTH OF THE NEXT STRING IS RETURNED 

C AS THE FUNCTION'S VALUE. THE SIZE OF THE CHAR ARRAY NEXTST IS NXTLEN. 

C 

FUNCTION NXTSTR ( STR , LEN , ST ART , NEXTST , NXTLEN ) 

INTEGER NXTSTR 

LOGICAL* 1 STR (120), NEXTST (120) 

INTEGER LEN, START, NXTLEN 
C 

INTEGER I, STRPTR 
LOGICAL* 1 CHAREQ , CHARPT 
C 

DOWHILE (( START. LE. LEN). AND. CHAREQ (STR ( START) , ' ') ) 

START = START + 1 
ENDDO 
STRPTR = 1 

DOWHILE ( (START. LE. LEN). AND. (. NOT. (CHAREQ (STR (START) , ' '))) ) 

NEXTST (STRPTR) = STR (START) 

START = START + 1 
STRPTR = STRPTR + 1 
ENDDO 
I = STRPTR 

DOWHILE ( I. LE. NXTLEN) 

NEXTST ( I ) = CHARPT (' ') 

1 = 1+1 
ENDDO 

NXTSTR = STRPTR - 1 
RETURN 
END 
C 

C STREQ ( STR 1 , STR2 , LEN ) RETURNS 1 IF THE STRING IN CHAR ARRAY STR1 IS 
C EQUIVALENT TO THE STRING IN CHARACTER ARRAY STR 2. 

C IT RETURNS 0 OTHERWISE. ARRAYS STR 1. AND STR2 ARE OF SIZE LEN. 

C 

FUNCTION STREQ(STR1 ,STR2,LEN) 

LOGICAL* 1 STREQ, STR1 (120) , STR2(120) 

INTEGER LEN 
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531: C 

532: INTEGER I, START, II, 12, NXTSTR 

533: LOGICAL* 1 CHAREQ 

534: LOGICAL* 1 WORD1(120), WORD2(;20) 

535: C 

536: START = 1 

537: II = NXTSTR (STR1,LEN, START, WORD 1,1 20) 

538: START = 1 

539: 12 = NXTSTR(STR2, LEN, START, W0RD2, 120) 

540: I = 1 

541: DOWHILE ( (I.LE.I1 ) .AND. (I.LE.I2) .AND. CHAREQ(W0RD1 (I) ,W0RD2(I) ) 

542: X .AND.(.N0T.(CHAREQ(W0RD1(I),' '))) 

543: X .AND. ( .NOT. (CHAREQ(W0RD2(I) , ' '))) ) 

544: 1=1+1 

545: ENDDO 

546: .IF (((I.GT.I1) .AND. (I.GT.I2) ) .OR. 

547: X ((CHAREQ(WORDKI),' ') ) .AND. (CHAREQ(W0RD2(I) , ' '))) ) 

548: STREQ = 1 

549: ELSE 

550: STREQ = 0 

551: ENDIF 

552: RETURN 

553: END 

554: C 

555: C W0RDEQ(W0RD1,PTR2) RETURNS 1 IF THE KEYWORD IN W0RD(PTR2 , J) , J= 1 , 12 IS 
556: C EQUIVALENT TO THE WORD IN CHARACTER ARRAY W0RD1 . 

557: C IT RETURNS 0 OTHERWISE. 

558: C 
559: 

560 : 

561: 

562: 

563: C 
564: 

565: 

566: 

567: C 
568: 

569: 

570: C 
571: 

572: 

573: 700 
574: 

575: 

576: 

577: 

578: 

579: 

580 : 


FUNCTION WORDEQ ( WORD 1 ,PTR2) 

LOGICAL* 1 WORDEQ 
LOGICAL* 1 W0RD1 (12) 

INTEGER PTR2 

LOGICAL* 1 WORD (500, 12) , REFKEY( 1000, 3) 

INTEGER NUMWDS, NUMREF, PTR(500), NEXT (1000) 

COMMON /WORDS/ WORD, REFKEY, NUMWDS, NUMREF, PTR, NEXT 

LOGICAL* 1 W0RD2 (12), STREQ 
INTEGER I 

DO 700 I = 1, 12 

W0RD2(I) = W0RD(PTR2,I) 

CONTINUE 

.IF (STREQ ( WORD 1,W0RD2, 12)) 

WORDEQ = 1 
ELSE 

WORDEQ = 0 
ENDIF 
RETURN 
END 
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PROGRAM 3 FAULTS 


1. Three-character words are treated as keywords. 

2. The key 'zzz' is not recognized. 

2. a Any key greater than 'zzz' causes loop. 

3. If action ADD occurs with key already in file, the 
program acts like REPLACE; the update record is not 
skipped. 

4. If REPLACE key is not found, the program acts like 
ADD; the update record is not skipped. 

5. A maximum of 500 keywords and 1000 reference keys 
are allowed. 

6. Greater than 2 transactions for the same master 
record produces incorrect results. 

7. Keywords greater than 12 characters are truncated 
and not distinguished. 

8. UPDATE transaction with column 80 not an ' R' 
produces same result as ADD. 

9. Keyword indices appear in the opposite order from 
that shown in specifications. 

10. No check is made for unique keys in the master file. 

11. Punctuation is made a part of the keyword. 

12. A word appearing twice in a title gets two 
cross-reference entries. 


a Alternate manifestation of this error. 
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APPENDIX B 


DATA SUMMARY 


This appendix contains a set of tables summarizing the re- 
sults obtained from those subjects who passed the initial 
screening and actually participated in the experiment. The 
original materials provided by each subject are reproduced 
in the Data Supplement. 
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SUBJECT 02 


EXPERIENCE LEVEL : Junior 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 44 

Percent Estimated Faults: 20 

Hours To Detect: 4.75 

Hours To Correct: 1.25 

Hours per Fault: 1.5 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 86 

Percent Estimated Faults: 75 

Hours To Detect: 5 

Hours To Correct: 2.5 

Hours per Fault: 1.25 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 17 

Percent Estimated Faults: 30 

Hours To Detect: 6.75 

Hours To Correct: 1 

Hours per Fault: 3.88 


9846 
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SUBJECT 03 


EXPERIENCE LEVEL ; Intermediate 
COMPUTER : IBM 

PROGRAM I : 

Verification Method: Reading 

Percent Faults Found: 56 

Percent Estimated Faults: 90 

Hours To Detect: 3 

Hours To Correct: 3 

Hours per Fault: 1.2 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 100 

Percent Estimated Faults: 95 

Hours To Detect: 2.5 

Hours To Correct: 0.75 

Hours per Fault: 0.46 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 42 • 

Percent Estimated Faults: 70 

Hours To Detect: 3.5 

Hours To Correct: 3 

Hours per Fault: 1.3 


9846 
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SUBJECT 04 


EXPERIENCE LEVEL ; Junior 
COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 33 

Percent Estimated Faults: 100 

Hours To Detect: 0.75 

Hours To Correct: 1 

Hours per Fault: 0.58 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults: 100 

Hours To Detect: 2 

Hours To Correct: 0.5 

Hours per Fault: 0.36 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 50 

Percent Estimated Faults: 70 

Hours To Detect: 4 

Hours To Correct: 4 

Hours per Fault: 1.33 


9846 
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SUBJECT 05 


EXPERIENCE LEVEL : Intermediate 

COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Reading 

Percent Faults Found: 78 

Percent Estimated Faults: 100 

Hours To Detect: 3.5 

Hours To Correct: 4.25 

Hours per Fault: 1.11 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 43 

Percent Estimated Faults: 100 

Hours To Detect: 4 

Hours To Correct: 3.5 

Hours per Fault: 2.5 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 90 

Percent Estimated Faults: 43 

Hours To Detect: 6.25 

Hours To Correct: 1.75 

Hours per Fault: 1.60 


9846 
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EXPERIENCE LEVEL : Junior 

COMPUTER ; VAX 
PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found; 67 
Percent Estimated Faults: 75 

Hours To Detect: 1.5 

Hours To Correct: 1 

Hours per Fault: 0.42 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 29 

Percent Estimated Faults: 50 

Hours To Detect: 0.75 

Hours To Correct: 0.25 

Hours per Fault: 0.5 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 25 

Percent Estimated Faults: 50 

Hours To Detect: 1.5 

Hours To Correct: 0.25 

Hours per Fault: 0.58 


9846 
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EXPERIENCE LEVEL ; Junior 
COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Reading 

Percent Faults Found: 44 

Percent Estimated Faults: 93 

Hours To Detect: 2.5 

Hours To Correct: 3.5 

Hours per Fault: 1.5 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 57 

Percent Estimated Faults: 85 

Hours To Detect: 4.5 

Hours To Correct: 1 

Hours per Fault: 1.38 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 17 

Percent Estimated Faults: 80 

Hours To Detect: 3.75 

Hours To Correct: 0.75 

Hours per Fault: 2.25 


9846 
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EXPERIENCE LEVEL ; Advanced 
COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 78 

Percent Estimated Faults: 50 

Hours To Detect: 5.5 

Hours To Correct: 2.5 

Hours per Fault: 1.14 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults:- 100 
Hours To Detect: 1.25 

Hours To Correct: 1.5 

Hours per Fault: 0.39 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 50 

Percent Estimated Faults: 80 

Hours To Detect: 3 

Hours To Correct: 2 

Hours per Fault: 0.83 


9846 
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EXPERIENCE LEVEL : Junior 

COMPUTER ; VAX 
PROGRAM 1 ; 

Verification Method: Reading 

Percent Faults Found: 67 

Percent Estimated Faults: 75 

Hours To Detect: 2.25 

Hours To Correct: 0.5 

Hours per Fault: 0.46 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 86 

Percent Estimated Faults: 100 

Hours To Detect: 1.5 

Hours To Correct: 0.25 

Hours per Fault: 0.29 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 42 

Percent Estimated Faults: 90 

Hours To Detect: 1.5 

Hours To Correct: 1.5 

Hours per Fault: 0.6 


9846 
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EXPERIENCE LEVEL ; Junior 
COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 56 

Percent Estimated Faults: 80 

Hours To Detect: 3 

Hours To Correct: 2 

Hours per Fault: 1 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 43 

Percent Estimated Faults: 75 

Hours To Detect: 3.5 

Hours To Correct: 3 

Hours per Fault: 2.17 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 17 

Percent Estimated Faults: 10 

Hours To Detect: 6 

Hours To Correct: 1 

Hours per Fault: 3.5 


9846 
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EXPERIENCE LEVEL : Junior 

COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 11 

Percent Estimated Faults: 60 

Hours To Detect: 1 

Hours To Correct: 4 

Hours per Fault: 5 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 57 

Percent Estimated Faults: 80 

Hours To Detect: 1.5 

Hours To Correct: 2 

Hours per Fault: 0.88 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 0 

Percent Estimated Faults: 5 

Hours To Detect: 4.5 

Hours To Correct: 1.5 

Hours per Fault: 
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SUBJECT 14 


EXPERIENCE LEVEL : Junior 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Code Reading 

Percent Faults Found: 89 

Percent Estimated Faults: 90 

Hours To Detect: 4 

Hours To Correct: 1 

Hours per Fault: 0.63 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 43 

Percent Estimated Faults: 90 

Hours To Detect: 3 

Hours To Correct: 3.25 

Hours per Fault: 2.08 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 33 

Percent Estimated Faults: 80 

Hours To Detect: 4 

Hours To Correct: 3.5 

Hours per Fault: 1.88 
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SUBJECT 15 


EXPERIENCE LEVEL ; Intermediate 
COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 67 

Percent Estimated Faults: 75 

Hours To Detect: 4.5 

Hours To Correct: 3.5 

Hours per Fault: 1.33 

PROGRAM 2 ; 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults: 75 

Hours To Detect: 2.5 

Hours To Correct: 1 

Hours per Fault: 0.50 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 33 

Percent Estimated Faults: 60 

Hours To Detect: 3.5 

Hours To Correct: 2 

Hours per Fault: 1.38 
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SUBJECT 17 


EXPERIENCE LEVEL : Junior 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 33 

Percent Estimated Faults: 40 

Hours To Detect: 3 

Hours To Correct: 1.5 

Hours per Fault: 1.17 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 57 

Percent Estimated Faults: 60 

Hours To Detect: 3 

Hours To Correct: 0.5 

Hours per Fault: 0.88 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 25 

Percent Estimated Faults: 30 

Hours To Detect: 5 

Hours To Correct: 1 

Hours per Fault: 2 
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SUBJECT 21 


EXPERIENCE LEVEL : Junior 

COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 33 

Percent Estimated Faults: 90 

Hours To Detect: 1.25 

Hours To Correct: 1.5 

Hours per Fault: 0.92 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 71 

Percent Estimated Faults: 100 

Hours To Detect: 1.25 

Hours To Correct: 0.5 

Hours per Fault: 0.35 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 17 

Percent Estimated Faults: 20 

Hours To Detect: 5 

Hours To Correct: 0 

Hours per Fault: 2.5 


SUBJECT 23 


EXPERIENCE LEVEL : Intermediate 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 11 

Percent Estimated Faults: 2 

Hours To Detect: 0.5 

Hours To Correct: 0.5 

Hours per Fault: 1 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 71 

Percent Estimated Faults: 90 

Hours To Detect: 0.5 

Hours To Correct: 0.5 

Hours per Fault: 0.2 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 8 

Percent Estimated Faults: 60 

Hours To Detect: 3 

Hours To Correct: 1 

Hours per Fault: 4 
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SUBJECT 25 


EXPERIENCE LEVEL : Intermediate 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Reading 

Percent Faults Found: 44 

Percent Estimated Faults: 70 

Hours To Detect: 3.5 

Hours To Correct: 2 

Hours per Fault: 1.25 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 100 

Percent Estimated Faults: 100 

Hours To Detect: 3.25 

Hours To Correct: 0.25 

Hours per Fault: 0.5 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 0 

Percent Estimated Faults: 0 

Hours To Detect: 0.75 

Hours To Correct: 0.5 

Hours per Fault: 
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SUBJECT 26 


EXPERIENCE LEVEL : Intermediate 

COMPUTER ; VAX 
PROGRAM 1 ; 

Verification Method: Structural 

Percent Faults Found; 22 
Percent Estimated Faults: 50 

Hours To Detect: 2 

Hours To Correct: 0.75 

Hours per Fault: 1.38 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 86 

Percent Estimated Faults: 75 

Hours To Detect: 3.25 

Hours To Correct: 0.75 

Hours per Fault: 0.67 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 25 

Percent Estimated Faults: 75 

Hours To Detect: 2 

Hours To Correct: 0.5 

Hours per Fault: 0.83 
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SUBJECT 28 


EXPERIENCE LEVEL : Intermediate 

COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 44 

Percent Estimated Faults:' 95 
Hours To Detect: 0.75 

Hours To Correct: 2 

Hours per Fault: 0.69 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults: 100 

Hours To Detect: 0.75 

Hours To Correct: 0.5 

Hours per Fault: 0.18 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 17 

Percent Estimated Faults: 60 

Hours To Detect: 2 

Hours To Correct: 2.5 

Hours per Fault: 2.25 
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SUBJECT 29 


EXPERIENCE LEVEL : Advanced 

COMPUTER ; IBM 
PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 33 

Percent Estimated Faults: 80 

Hours To Detect: 2 

Hours To Correct: 2.25 

Hours per Fault: 1.42 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 43 

Percent Estimated Faults: 100 

Hours To Detect: 2.75 

Hours To Correct: 1.25 

Hours per Fault: 1.33 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 50 

Percent Estimated Faults: 90 

Hours To Detect: 2 

Hours To Correct: 1.5 

Hours per Fault: 0.88 
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SUBJECT 30 


EXPERIENCE LEVEL : Intermediate 

COMPUTER i VAX 
PROGRAM 1 : 

Verification Method: Reading 

Percent Faults Found: 44 

Percent Estimated Faults: 81 

Hours To Detect: 3 

Hours To Correct: 0.5 

Hours per Fault: 0.88 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 86 

Percent Estimated Faults: 90 

Hours To Detect: 2.25 

Hours To Correct: 0.75 

Hours per Fault: 0.5 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 25 

Percent Estimated Faults: 80 

Hours To Detect: 2.5 

Hours To Correct: 1.75 

Hours per Fault: 1.42 
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SUBJECT 35 


EXPERIENCE LEVEL : Advanced 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Reading 

Percent Faults Found: 67 

Percent Estimated Faults: 85 

Hours To Detect: 2 

Hours To Correct: 2 

Hours per Fault: 0.67 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 100 

Percent Estimated Faults: 85 

Hours To Detect: 3.5 

Hours To Correct: 1 

Hours per Fault: 0.64 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 42 

Percent Estimated Faults: 75 

Hours To Detect: 2 

Hours To Correct: 3 

Hours per Fault: 1 
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SUBJECT 36 


EXPERIENCE LEVEL : Intermediate 

COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 33 

Percent Estimated Faults: 50 

Hours To Detect: 0.75 

Hours To Correct: 2 

Hours per Fault: 0.92 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults: 90 

Hours To Detect: 0.75 

Hours To Correct: 0.5 

Hours per Fault: 0.18 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 25 

Percent Estimated Faults: 75 

Hours To Detect: 2 

Hours To Correct: 2 

Hours per Fault: 1.33 
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SUBJECT 37 


EXPERIENCE LEVEL : Junior 

COMPUTER : VAX 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 44 

Percent Estimated Faults: 80 

Hours To Detect: 2 

Hours To Correct: 2 

Hours per Fault: 1 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 86 

Percent Estimated Faults: 95 

Hours To Detect: 0.75 

Hours To Correct: 0.5 

Hours per Fault: 0.21 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 33 

Percent Estimated Faults: 90 

Hours To Detect: 6 

Hours To Correct: 1.5 

Hours per Fault: 1.88 
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SUBJECT 38 


EXPERIENCE LEVEL : Advanced 

COMPUTER ; IBM 
PROGRAM 1 : 

Verification Method: Reading 

Percent Faults Found; 67 
Percent Estimated Faults: 90 

Hours To Detect: 6.75 

Hours To Correct: 1.5 

Hours per Fault: 1.38 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 57 

Percent Estimated Faults: 90 

Hours To Detect: 2.75 

Hours To Correct: 1.5 

Hours per Fault: 1.06 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 17 

Percent Estimated Faults: 60 

Hours To Detect: 5 

Hours To Correct: 1 

Hours per Fault: 3 
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SUBJECT 40 


EXPERIENCE LEVEL : Intermediate 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Functional 

Percent Faults Found: 44 

Percent Estimated Faults: 70 

Hours To Detect: 1.5 

Hours To Correct: 1.5 

Hours per Fault: 0.75 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 71 

Percent Estimated Faults: 100 

Hours To Detect: 2 

Hours To Correct: 0.5 

Hours per Fault: 0.5 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 33 

Percent Estimated Faults: 51 

Hours To Detect: 3 

Hours To Correct: 1 

Hours per Fault: 1 
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SUBJECT 41 


EXPERIENCE LEVEL : Junior 

COMPUTER ; VAX 
PROGRAM 1 ; 

Verification Method: Reading 

Percent Faults Found: 56 

Percent Estimated Faults: 100 

Hours To Detect: 3.5 

Hours To Correct: 1 

Hours per Fault: 0.9 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 57 

Percent Estimated Faults: 60 

Hours To Detect: 2.25 

Hours To Correct: 1.5 

Hours per Fault: 0.94 

PROGRAM 3 : 

Verification Method: Structural 

Percent Faults Found: 25 

Percent Estimated Faults: 80 

Hours To Detect: 2.75 

Hours To Correct: 1.5 

Hours per Fault: 1.42 


9846 


B- 27 


SUBJECT 42 


EXPERIENCE LEVEL : Advanced 

COMPUTER ; VAX 
PROGRAM 1 ; 

Verification Method: Structural 

Percent Faults Found: 44 

Percent Estimated Faults: 90 

Hours To Detect: 2 

Hours To Correct: 0.5 

Hours per Fault: 0.63 

PROGRAM 2 : 

Verification Method: Functional 

Percent Faults Found: 57 

Percent Estimated Faults: 100 

Hours To Detect: 3.5 

Hours To Correct: 1.5 

Hours per Fault: 1.25 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 33 

Percent Estimated Faults: 50 

Hours To Detect: 2.5 

Hours To Correct: 0.5 

Hours per Fault: 0.75 
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SUBJECT 43 


EXPERIENCE LEVEL : Intermediate 

COMPUTER ; IBM 
PROGRAM 1 ; 

Verification Method: Functional 

Percent Faults Found: 44 

Percent. Estimated Faults: 85 

Hours To Detect: 1.5 

Hours To Correct: 3 

Hours per Fault: 1.13 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 14 

Percent Estimated Faults: 90 

Hours To Detect: 0.75 

Hours To Correct: 0.75 

Hours per Fault: 1.5 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 25 

Percent Estimated Faults: 60 

Hours To Detect: 3 

Hours To Correct: 0.5 

Hours per Fault: 1.17 
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SUBJECT 45 


EXPERIENCE LEVEL : Advanced 

COMPUTER ; VAX 
PROGRAM 1 ; 

Verification Method: Functional 

Percent Faults Found: 44 

Percent Estimated Faults: 90 

Hours To Detect: 1.5 

Hours To Correct: 2.25 

Hours per Fault: 0.94 

PROGRAM 2 : 

Verification Method: Structural 

Percent Faults Found: 86 

Percent Estimated Faults: 100 

Hours To Detect: 1.5 

Hours To Correct: 0.5 

Hours per Fault: 0.33 

PROGRAM 3 : 

Verification Method: Reading 

Percent Faults Found: 42 

Percent Estimated Faults: 80 

Hours To Detect: 3.25 

Hours To Correct: 0.75 

Hours per Fault: 0.8 
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SUBJECT 47 


EXPERIENCE LEVEL ; Advanced 
COMPUTER ; VAX 
PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 44 

Percent Estimated Faults: 83 

Hours To Detect: 1.25 

Hours To Correct: 2.5 

Hours per Fault: 0.94 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults: 100 

Hours To Detect: 1.5 

Hours To Correct: 1.25 

Hours per Fault: 0.39 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 50 

Percent Estimated Faults: 70 

Hours To Detect: 7.25 

Hours To Correct: 3.5 

Hours per Fault: 1.79 
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SUBJECT 48 


EXPERIENCE LEVEL ; Advanced 
COMPUTER ; IBM 
PROGRAM 1 : 

Verification Method; Structural 
Percent Faults Found; 33 
Percent Estimated Faults; 50 
Hours To Detect; 1.25 
Hours To Correct: 0.25 

Hours per Fault: 0.5 

PROGRAM 2 ; 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults: 90 

Hours To Detect: 1 

Hours To Correct: 0.25 

Hours per Fault: 0.18 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 33 

Percent Estimated Faults: 80 

Hours To Detect: 3.75 

Hours To Correct: 2.25 

Hours per Fault: 1.56 
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SUBJECT 50 


EXPERIENCE LEVEL : Junior 

COMPUTER : IBM 

PROGRAM 1 : 

Verification Method: Structural 

Percent Faults Found: 11 

Percent Estimated Faults: 100 

Hours To Detect: 2 

Hours To Correct: 1 

Hours per Fault: 3 

PROGRAM 2 : 

Verification Method: Reading 

Percent Faults Found: 100 

Percent Estimated Faults: 100 

Hours To Detect: 0.5 

Hours To Correct: 2 

Hours per Fault: 0.36 

PROGRAM 3 : 

Verification Method: Functional 

Percent Faults Found: 17 

Percent Estimated Faults: 80 

Hours To Detect: 3 

Hours To Correct: 2 

Hours per Fault: 2.5 
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APPENDIX C - FAULTS FOUND 


Figures C-l through C-3 show the distribution of faults 
found by subjects during the experiment for the three test 
programs. The subject identification number and verifica- 
tion technique applied are also indicated in the figures. 
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